{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic 回归——Rental Listing Inquiries\n",
    "纽约市公寓用户感兴趣程度预测"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rental Listing Inquiries数据集是Kaggle平台上的一个分类竞赛任务，需要根据公寓的特征来预测其受欢迎程度（用户感兴趣程度分为高、中、低三类）。其中房屋的特征x共有14维，响应值y为用户对该公寓的感兴趣程度。评价标准为logloss。\n",
    "\n",
    "竞赛链接：https://www.kaggle.com/c/two-sigma-connect-rental-listing-inquiries\n",
    "\n",
    "由于房屋属性类型各异，对原有属性进行特征工程请见FE_RentListingInqueries.ipynb\n",
    "这里直接读取编码后的数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 首先 import 必要的模块\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "#竞赛的评价指标为logloss\n",
    "from sklearn.metrics import log_loss  \n",
    "\n",
    "from matplotlib import pyplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>price</th>\n",
       "      <th>price_bathrooms</th>\n",
       "      <th>price_bedrooms</th>\n",
       "      <th>room_diff</th>\n",
       "      <th>room_num</th>\n",
       "      <th>Year</th>\n",
       "      <th>...</th>\n",
       "      <th>walk</th>\n",
       "      <th>walls</th>\n",
       "      <th>war</th>\n",
       "      <th>washer</th>\n",
       "      <th>water</th>\n",
       "      <th>wheelchair</th>\n",
       "      <th>wifi</th>\n",
       "      <th>windows</th>\n",
       "      <th>work</th>\n",
       "      <th>interest_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.5</td>\n",
       "      <td>3</td>\n",
       "      <td>40.7145</td>\n",
       "      <td>-73.9425</td>\n",
       "      <td>3000</td>\n",
       "      <td>1200.0</td>\n",
       "      <td>750.000000</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>4.5</td>\n",
       "      <td>2016</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>40.7947</td>\n",
       "      <td>-73.9667</td>\n",
       "      <td>5465</td>\n",
       "      <td>2732.5</td>\n",
       "      <td>1821.666667</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>40.7388</td>\n",
       "      <td>-74.0018</td>\n",
       "      <td>2850</td>\n",
       "      <td>1425.0</td>\n",
       "      <td>1425.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>40.7539</td>\n",
       "      <td>-73.9677</td>\n",
       "      <td>3275</td>\n",
       "      <td>1637.5</td>\n",
       "      <td>1637.500000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>40.8241</td>\n",
       "      <td>-73.9493</td>\n",
       "      <td>3350</td>\n",
       "      <td>1675.0</td>\n",
       "      <td>670.000000</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 225 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   bathrooms  bedrooms  latitude  longitude  price  price_bathrooms  \\\n",
       "0        1.5         3   40.7145   -73.9425   3000           1200.0   \n",
       "1        1.0         2   40.7947   -73.9667   5465           2732.5   \n",
       "2        1.0         1   40.7388   -74.0018   2850           1425.0   \n",
       "3        1.0         1   40.7539   -73.9677   3275           1637.5   \n",
       "4        1.0         4   40.8241   -73.9493   3350           1675.0   \n",
       "\n",
       "   price_bedrooms  room_diff  room_num  Year       ...        walk  walls  \\\n",
       "0      750.000000       -1.5       4.5  2016       ...           0      0   \n",
       "1     1821.666667       -1.0       3.0  2016       ...           0      0   \n",
       "2     1425.000000        0.0       2.0  2016       ...           0      0   \n",
       "3     1637.500000        0.0       2.0  2016       ...           0      0   \n",
       "4      670.000000       -3.0       5.0  2016       ...           0      0   \n",
       "\n",
       "   war  washer  water  wheelchair  wifi  windows  work  interest_level  \n",
       "0    0       0      0           0     0        0     0               1  \n",
       "1    0       0      0           0     0        0     0               2  \n",
       "2    0       0      0           0     0        0     0               0  \n",
       "3    0       0      0           0     0        0     0               2  \n",
       "4    1       0      0           0     0        0     0               2  \n",
       "\n",
       "[5 rows x 225 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = pd.read_csv(\"RentListingInquries_FE_train.csv\")\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 49352 entries, 0 to 49351\n",
      "Columns: 225 entries, bathrooms to interest_level\n",
      "dtypes: float64(7), int64(218)\n",
      "memory usage: 84.7 MB\n"
     ]
    }
   ],
   "source": [
    "train.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>listing_id</th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>price</th>\n",
       "      <th>price_bathrooms</th>\n",
       "      <th>price_bedrooms</th>\n",
       "      <th>room_diff</th>\n",
       "      <th>room_num</th>\n",
       "      <th>...</th>\n",
       "      <th>virtual</th>\n",
       "      <th>walk</th>\n",
       "      <th>walls</th>\n",
       "      <th>war</th>\n",
       "      <th>washer</th>\n",
       "      <th>water</th>\n",
       "      <th>wheelchair</th>\n",
       "      <th>wifi</th>\n",
       "      <th>windows</th>\n",
       "      <th>work</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7142618</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>40.7185</td>\n",
       "      <td>-73.9865</td>\n",
       "      <td>2950</td>\n",
       "      <td>1475.000000</td>\n",
       "      <td>1475.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7210040</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>40.7278</td>\n",
       "      <td>-74.0000</td>\n",
       "      <td>2850</td>\n",
       "      <td>1425.000000</td>\n",
       "      <td>950.000000</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7103890</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>40.7306</td>\n",
       "      <td>-73.9890</td>\n",
       "      <td>3758</td>\n",
       "      <td>1879.000000</td>\n",
       "      <td>1879.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7143442</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>40.7109</td>\n",
       "      <td>-73.9571</td>\n",
       "      <td>3300</td>\n",
       "      <td>1650.000000</td>\n",
       "      <td>1100.000000</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6860601</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>40.7650</td>\n",
       "      <td>-73.9845</td>\n",
       "      <td>4900</td>\n",
       "      <td>1633.333333</td>\n",
       "      <td>1633.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 225 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   listing_id  bathrooms  bedrooms  latitude  longitude  price  \\\n",
       "0     7142618        1.0         1   40.7185   -73.9865   2950   \n",
       "1     7210040        1.0         2   40.7278   -74.0000   2850   \n",
       "2     7103890        1.0         1   40.7306   -73.9890   3758   \n",
       "3     7143442        1.0         2   40.7109   -73.9571   3300   \n",
       "4     6860601        2.0         2   40.7650   -73.9845   4900   \n",
       "\n",
       "   price_bathrooms  price_bedrooms  room_diff  room_num  ...   virtual  walk  \\\n",
       "0      1475.000000     1475.000000        0.0       2.0  ...         0     0   \n",
       "1      1425.000000      950.000000       -1.0       3.0  ...         0     0   \n",
       "2      1879.000000     1879.000000        0.0       2.0  ...         0     0   \n",
       "3      1650.000000     1100.000000       -1.0       3.0  ...         0     0   \n",
       "4      1633.333333     1633.333333        0.0       4.0  ...         0     0   \n",
       "\n",
       "   walls  war  washer  water  wheelchair  wifi  windows  work  \n",
       "0      0    0       0      0           0     0        0     0  \n",
       "1      0    1       0      0           0     0        0     0  \n",
       "2      0    0       0      0           0     0        0     0  \n",
       "3      0    0       0      0           1     0        0     0  \n",
       "4      0    1       0      0           0     0        0     0  \n",
       "\n",
       "[5 rows x 225 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = pd.read_csv(\"RentListingInquries_FE_test.csv\")\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 74659 entries, 0 to 74658\n",
      "Columns: 225 entries, listing_id to work\n",
      "dtypes: float64(7), int64(218)\n",
      "memory usage: 128.2 MB\n"
     ]
    }
   ],
   "source": [
    "test.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 将类别字符串变成数字\n",
    "# drop ids and get labels\n",
    "y_train = train['interest_level']   \n",
    "X_train = train.drop([\"interest_level\"], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_Id = test['listing_id']\n",
    "X_test = test.drop([\"listing_id\"], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 数据标准化\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# 初始化特征的标准化器\n",
    "ss_X = StandardScaler()\n",
    "\n",
    "# 分别对训练和测试数据的特征进行标准化处理\n",
    "X_train = ss_X.fit_transform(X_train)\n",
    "X_test = ss_X.transform(X_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### default Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lr= LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "F:\\Soft\\AI\\Anaconda2\\lib\\site-packages\\sklearn\\cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n",
      "F:\\Soft\\AI\\Anaconda2\\lib\\site-packages\\sklearn\\linear_model\\base.py:340: RuntimeWarning: overflow encountered in exp\n",
      "  np.exp(prob, prob)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logloss of each fold is:  [ 0.68933112  0.68560022  0.68066906  0.67145482  0.68216311]\n",
      "cv logloss is: 0.681843666539\n"
     ]
    }
   ],
   "source": [
    "# 交叉验证用于评估模型性能和进行参数调优（模型选择）\n",
    "#分类任务中交叉验证缺省是采用StratifiedKFold\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "loss = cross_val_score(lr, X_train, y_train, cv=5, scoring='neg_log_loss')\n",
    "print 'logloss of each fold is: ',-loss\n",
    "print'cv logloss is:', -loss.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 正则化的 Logistic Regression及参数调优"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "logistic回归的需要调整超参数有：C（正则系数，一般在log域（取log后的值）均匀设置候选参数）和正则函数penalty（L2/L1） \n",
    "目标函数为：J = sum(logloss(f(xi), yi)) + （1/C）* penalty \n",
    "\n",
    "在sklearn框架下，不同学习器的参数调整步骤相同：\n",
    "设置候选参数集合\n",
    "调用GridSearchCV\n",
    "调用fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "F:\\Soft\\AI\\Anaconda2\\lib\\site-packages\\sklearn\\model_selection\\_split.py:2026: FutureWarning: From version 0.21, test_size will always complement train_size unless both are specified.\n",
      "  FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "# 训练样本6w+，交叉验证太慢，用train_test_split估计模型性能\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train_part, X_val, y_train_part, y_val = train_test_split(X_train, y_train, train_size = 0.33,random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "def fit_grid_point_LR(penalty, C, X_train, y_train, X_val, y_val):\n",
    "    \n",
    "    # 在训练集训练LR\n",
    "    LR = LogisticRegression(penalty=penalty, C=C)\n",
    "    LR.fit(X_train, y_train)\n",
    "    \n",
    "    # 在训练集和校验集上测试\n",
    "    y_train_pred = LR.predict_proba(X_train)\n",
    "    y_val_pred = LR.predict_proba(X_val)\n",
    "    logloss_val = log_loss(y_val,y_val_pred)\n",
    "    logloss_train = log_loss(y_train, y_train_pred)\n",
    "    \n",
    "    print(\"logloss on test: %f and on train: %f with C = %f and penalty = %s\"%(logloss_val, logloss_train, C, penalty) )\n",
    "    return logloss_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logloss on test: 0.810481 and on train: 0.810799 with C = 0.001000 and penalty = l1\n",
      "logloss on test: 0.774072 and on train: 0.766717 with C = 0.001000 and penalty = l2\n",
      "logloss on test: 0.734006 and on train: 0.732657 with C = 0.010000 and penalty = l1\n",
      "logloss on test: 0.727626 and on train: 0.706260 with C = 0.010000 and penalty = l2\n",
      "logloss on test: 0.689003 and on train: 0.670205 with C = 0.100000 and penalty = l1\n",
      "logloss on test: 0.718411 and on train: 0.675056 with C = 0.100000 and penalty = l2\n",
      "logloss on test: 0.710631 and on train: 0.656309 with C = 1.000000 and penalty = l1\n",
      "logloss on test: 0.734140 and on train: 0.654931 with C = 1.000000 and penalty = l2\n",
      "logloss on test: 0.749422 and on train: 0.653840 with C = 10.000000 and penalty = l1\n",
      "logloss on test: 0.758580 and on train: 0.650102 with C = 10.000000 and penalty = l2\n",
      "logloss on test: 0.767878 and on train: 0.653497 with C = 100.000000 and penalty = l1\n",
      "logloss on test: 0.773679 and on train: 0.648849 with C = 100.000000 and penalty = l2\n",
      "logloss on test: 0.771619 and on train: 0.653417 with C = 1000.000000 and penalty = l1\n",
      "logloss on test: 0.782065 and on train: 0.648748 with C = 1000.000000 and penalty = l2\n"
     ]
    }
   ],
   "source": [
    "#需要调优的参数\n",
    "penaltys = ['l1','l2']\n",
    "Cs = [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "\n",
    "logloss_s = []\n",
    "for i, OneC in enumerate(Cs):\n",
    "    for j, onePenalty in enumerate(penaltys):\n",
    "        tmp = fit_grid_point_LR(onePenalty, OneC, X_train_part, y_train_part, X_val, y_val)\n",
    "        logloss_s.append(tmp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd4VNXWwOHfSgeSEEoo6XQILUAg\nFAFRkC4iojS7clVseG3XXj4UrwULNmxcFUFEEFQQUEB6IKETOlJCb6GF9P39cQaNGJiQzGQmk/U+\nTx4zM+fsswaTrNnlrC3GGJRSSqlL8XJ1AEoppdyfJgullFJ2abJQSilllyYLpZRSdmmyUEopZZcm\nC6WUUnZpslBKKWWXJgullFJ2abJQSilll4+rA3CUqlWrmpiYGFeHoZRSpUpycvJRY0yoveM8JlnE\nxMSQlJTk6jCUUqpUEZHdhTlOh6GUUkrZpclCKaWUXZoslFJK2eUxcxZKqbItOzub1NRUMjIyXB2K\nWwoICCAiIgJfX98ina/JQinlEVJTUwkKCiImJgYRcXU4bsUYw7Fjx0hNTaVWrVpFakOHoZRSHiEj\nI4MqVapooiiAiFClSpVi9bo0WSilPIYmiosr7r9NmU8WJ9OzGTN3K1sPnXZ1KEop5bbKfLLIM4YP\nf9/B18sLdV+KUkrZlZCQQFxcHFFRUYSGhhIXF0dcXBy7du26rHamTp3K5s2bC3wtJyeHkJAQAPLy\n8ujevTshISFcd911xQ2/QGV+grtSBT/6NK3J1FX7eKJHQyr4l/l/EqVUMSUmJgIwfvx4kpKSGDt2\nbJHamTp1Kl5eXjRs2PCSx4kIjz/+OKdPn2b8+PFFupY9Zb5nATC0bTRnMnOYsXa/q0NRSnm4WbNm\n0a5dO1q2bMlNN93E2bNnAXjssceIjY2lWbNmPPHEEyxatIiZM2cycuRIu70SEeHqq68mMDDQaXHr\nx2igZVQIDWsE8fXy3QxqHamTZEqVci/+uJGU/acc2mZsWDDP921crDYOHz7M6NGj+e233yhfvjyj\nRo3inXfe4c4772TmzJls3LgRESEtLY2QkBB69erFDTfc4LShpcuhPQusrDy0bTQb959ibepJV4ej\nlPJQS5cuJSUlhfbt2xMXF8eECRPYtWsXlStXxsvLi7vvvptp06ZRoUIFV4f6D9qzsOnfIpzRMzcx\nYflu4iJDXB2OUqoYitsDcBZjDD169OCrr776x2tJSUnMnTuXSZMm8eGHHzJnzpy/vb5r164/exj3\n338/t912W0mE/CdNFjaB/j70axHO1FWpPNM7lorli3ZLvFJKXUz79u156KGH2LlzJ7Vr1+bs2bPs\n37+fGjVqkJGRQZ8+fUhISCA2NhaAoKAgTp+2lvXHxMSwZs2aP9vKyckp0dh1GCqfoQlRZGTn8f2q\nVFeHopTyQNWrV+ezzz7jpptuonnz5rRv356tW7dy8uRJevfuTfPmzbnqqqt46623ABg8eDCvvPJK\noZbdtmvXjsGDBzN79mwiIiL47bffHBq7GGMc2qCrxMfHG0dsftT/gyWcOpfNr4901olupUqRTZs2\n0ahRI1eH4dYK+jcSkWRjTLy9c7VncYGhCdHsOHKW5TuPuzoUpZRyG5osLtCnWU0qlvNlQqLe0a2U\nUudpsrhAgK83N7SKYPbGgxw5nenqcJRSyi1osijAkIQosnMNk5P2ujoUpZRyC5osClAnNJD2daow\nccUecvM8YwGAUkoVh1OThYj0EJEtIrJdRJ4s4PUoEZkvIqtFZJ2I9Mr32n9s520Rke7OjLMgQxOi\nST1xjoXbjpT0pZVSyu04LVmIiDfwPtATiAUGi0jsBYc9A0w2xrQABgEf2M6NtT1uDPQAPrC1V2K6\nxVanaqA/E7R0uVLqMpV0ifLk5GTatm1LkyZNaNasGVOmTCnuW/gHZ97B3QbYbozZCSAik4B+QEq+\nYwwQbPu+InC+7Gs/YJIxJhP4Q0S229pb5sR4/8bPx4tBrSP5YMF29qWdIzykXEldWilVypV0ifLA\nwEAmTJhAnTp1SE1NJT4+nu7duxMUFFSk6xbEmcNQ4UD+GeJU23P5vQAME5FUYCbwwGWci4gMF5Ek\nEUk6csTxw0WD2kRigEkr9ji8baVU2eSMEuUNGjSgTp06AERERFClShWOHj3q0Lid2bMo6PbnC2eL\nBwPjjTFvikg74CsRaVLIczHGjAPGgXUHdzHj/YeISuXp0qAak1bu5cGr6+HrresBlCoVZj0JB9c7\nts0aTaHn6GI1URIlypcuXQpYtaQcyZl//VKByHyPI/hrmOm8O4HJAMaYZUAAULWQ55aIYW2jOHI6\nk19TDrni8kopD+LsEuX79u3jtttuY/z48Q4vV+TMnsVKoJ6I1AL2YU1YD7ngmD3A1cB4EWmElSyO\nADOAb0TkLSAMqAescGKsF9W5fjXCQ8rxdeJuejat6YoQlFKXq5g9AGdxZony88UIX3vtNVq3bu3w\n2J2WLIwxOSJyPzAb8AY+N8ZsFJGXgCRjzAzg38AnIjISa5jpNmNVNtwoIpOxJsNzgBHGmFxnxXop\n3l7C4DaRvDFnKzuPnKF2qPO2LVRKeTZnlSjPzMykX79+3HnnnfTv398psTt1EN4YM9MYU98YU8cY\nM8r23HO2RIExJsUY08EY09wYE2eMmZPv3FG28xoYY2Y5M057bmwdiY+XMFEnupVSxeCsEuUTJ05k\n6dKlfPbZZ38u012/3rFzNlqivJBGTFjFkh1HWf6fqwnwLdFbPpRShaAlyu3TEuUlYGhCFGnp2cxc\nf8DVoSilVInTZFFI7epUoXbVCkxI1KEopVTZo8mikESEIQlRJO8+waYDp1wdjlKqAJ4yrO4Mxf23\n0WRxGW5oFYG/j5dujKSUGwoICODYsWOaMApgjOHYsWMEBAQUuQ1n3mfhcULK+9GnWRjTVu3jyZ6N\nCPTXfz6l3EVERASpqak4o/SPJwgICCAiIqLI5+tfu8s0tG0U369KZfqafQxNiHZ1OEopG19fX2rV\nquXqMDyWDkNdphaRIcTWDObr5Xu0u6uUKjM0WVwmEWFo2yg2HTjF6r1prg5HKaVKhCaLIugXF04F\nP28mLNdltEqpskGTRREE+vvQv2U4P63bT1p6lqvDUUopp9NkUURD2kSTmZPHlORUV4eilFJOp8mi\niGLDgmkZFcI3iTrRrZTyfJosimFY22h2Hj3Lsp3HXB2KUko5lSaLYujVtCYh5X11olsp5fE0WRRD\ngK83N7SMYPbGgxw+neHqcJRSymk0WRTTkIQocvIM3yXpRLdSynNpsiim2qGBdKhbhW8S95CbpxPd\nSinPpMnCAYYmRLMv7Ry/bz3s6lCUUsopNFk4QLfY6oQG+etEt1LKY2mycABfby8GtY5k3pbDpJ5I\nd3U4SinlcJosHGRQmygEmLRir6tDUUoph9Nk4SDhIeW4qmE1Jq3cS1ZOnqvDUUqVBcbA0W2wZ7nT\nL+XUZCEiPURki4hsF5EnC3h9jIissX1tFZG0fK/9V0Q2isgmEXlXRMSZsTrC0IRojp7JZG7KIVeH\nopTyVGePwvopMH0EjGkCY+Nh5qNOv6zTdsoTEW/gfaAbkAqsFJEZxpiU88cYY0bmO/4BoIXt+/ZA\nB6CZ7eXFQGdggbPidYRO9UMJDynHhMTd9G5W09XhKKU8QfY52LMMdsyHnfPh4Hrr+YCKUKsTdHwE\n6nRxehjO3Fa1DbDdGLMTQEQmAf2AlIscPxh43va9AQIAP0AAX8DtP657ewlDEqJ4ffYWdhw5Q53Q\nQFeHpJQqbfLy4ND6v5LD7mWQmwlevhCZAFc9A7WvgrA48PIusbCcmSzCgfyzvalAQkEHikg0UAuY\nB2CMWSYi84EDWMlirDFmkxNjdZgb4yN5+9etfJO4h2f7xLo6HKVUaZC210oMO+bDH79Duq04abVY\naH0X1L4SotuDv+s+gDozWRQ0x3CxW5wHAVOMMbkAIlIXaARE2F6fKyKdjDEL/3YBkeHAcICoqCiH\nBF1coUH+dG9cgynJqTzWvQEBviWX+ZVSpUTGSdi1+K/ew7Ht1vOBNaBuN2tYqfaVEFTDlVH+jTOT\nRSoQme9xBLD/IscOAkbke9wfWG6MOQMgIrOAtsDfkoUxZhwwDiA+Pt5tam0MTYjmp3UH+GndAW5o\nFWH/BKWUZ8vNhtSkv3oP+5LB5IJvBYjpAPF3WgkitCG46VoeZyaLlUA9EakF7MNKCEMuPEhEGgCV\ngGX5nt4D3C0ir2L1UDoDbzsxVodqW7sydUIrMCFxtyYLpcoiY+DoVti5wEoOuxZD1mkQLwhrAVeM\ntJJDRBvw8XN1tIXitGRhjMkRkfuB2YA38LkxZqOIvAQkGWNm2A4dDEwyf99ubgpwFbAea+jqF2PM\nj86K1dFEhKEJ0bz0Uwob95+kcVhFV4eklHK2M0es5LBzvvXfU/us5yvVgmYDoXYXqNURylVyZZRF\nJp6yJWh8fLxJSkpydRh/OpmeTZtXfmVAqwhe6d/U1eEopRwtKx32LLX1HhZYK5gAAkKgdmcrOdTp\nApViXBikfSKSbIyJt3ecM4ehyrSK5X3p2zyM6av38VSvRgT66z+1UqVaXh4cXPvXpPSeRGtJq7ef\ntaT16uesBFGzeYkuaS0p+hfMiYa1jWZKcio/rN7HsLbRrg5HKXW5Tuz+a1hp5+9w7rj1fPUm0OZu\nKzlEtwO/Ci4NsyRosnCi5hEVaRwWzNfLdzM0IYpSULFEqbLtXBrsWvRX7+H4Tuv5oJpQv4c1rFSr\nMwRVd22cLqDJAmDbXOuGFwd/Ojg/0f3UtPWs2pNGq+jSObGllMfKyYLUlX/1HvYlg8mzLWm9AtoM\nt3oPoQ3cdklrSdFkcXQbTBhodSl7ve7w5vvFhfHKzE1MSNytyUIpd5CXB5t/gtVfW0tas89aS1rD\nW0HHR63eQ3h8qVnSWlI0WVStB23vheUfQKO+VmEuB6rg70P/FuF8m7SXZ3vHUqmC/gAq5RK5ObBx\nGix6A45shoqR0HyQlRxiOkK5EFdH6NZ0PwuAq56FynWskr+Zpx3e/NC2UWTl5PH9qlSHt62UsiMn\nC1Z9Be+3hql3AQLXfwoProE+b1kfEjVR2KXJAsCvPFz3oVXMa+5zDm++YY1g4qMrMSFxD3l5nnFf\ni1JuLzsDVnwC77WEGfeDXyDc+BXcu9S6Sc67dA+sGGM4fDqDNXvTSN59wunXK93/Wo4UlQDt74el\n70Gjax1eH35o2yhGfruWZTuP0aFuVYe2rZTKJ+ssJH1h/S6fOWiV1Oj9FtTrVqomqbNy8jh4MoPU\ntHT2p2Ww78Q59qWlsy/tnPU47dyfu3I2j6jI9PuvcGo8mizy6/I0bPkFZjxgffoICHZY0z2b1OSl\nH1OYkLhbk4VSzpBx0upJLP/AKvEd0xGuH2fNQ7phkjidkW37o59uSwRWAth3wkoIh09ncmGBjWpB\n/oSFlCM2LJhrYqsTFlKO8JByRFcp7/R4NVnk51vOGo76/BqY8wxc+67Dmg7w9WZgfCSfL/6Dw6cy\nqBYc4LC2lSrT0o/D8g8h8WPIPGmV+O70KES1dVlIxhiOnMlk34lzF00IpzJy/naOn7cXNUMCCA8p\nR8d61q6b4ZXKERFSjrCQctQMCcDfx3V3hmuyuFBka2j/ICx5G2KvhbpdHdb04DZRjFu4k29X7uWB\nq+s5rF2lyqTTh2DZWFj5mbX8tWEfK0mEtXD6pfMPEf0tIRQwRHReUICPlQBCytE6ptKfvYLzCaFq\noD9eXu7XAzpPk0VBrvwPbP0Fpj8A9y1z2EqJWlUrcEXdqkxcsYf7utTF241/MJRyWydTYcm7sOp/\nkJsFja+Hjv+G6o7bmfLCIaLU80nAzhBReKW/hojCK5UjrKKVDMIrlSM4wNdh8bmCJouC+AbAdR/A\np91g9tNw3fsOa3pY2yju+XoV8zcfpmts2SsZoFSRHf8DFo+BNd8ABpoNsvaFqFr3spvKyc1j4/5T\npJ6fND5xzhoesiWESw0RdaoXavUK3GiIqCRosriY8FZwxcOw6E2I7Qf1r3FIs1c3qk61IH8mJO7W\nZKFUYRzZAovegvXfWdVcW94CHR6CSkUrzpm48xjPz9jI5oN/3VN14RBRuC0JlJYhopKgyeJSOj8B\nW2bBjw/ahqOKX67D19uLQW2ieG/eNvYeTyeysvNXMShVKh1cDwvfgJTp4BMACfdA+wcguGbRmjuZ\nwSszNzFj7X7CQ8rxxsDmNA4L9oghopKgyeJSfPyt1VGfXAW//Af6f+SQZge1jmTsvG1MXLGHx3s0\ndEibSnmM1GRY+DpsnQV+QdZQU7sRUKFoS86zcvL4bPEfvDdvGzl5hgevrse9netQzs+zh40cTZOF\nPWFx1gqL31+zbtZr2Kv4TYaU46qG1ZmctJeHu9bHz0dvpFeKXUusJLFzvtWL7/K0VeCzGD36BVsO\n89KPKew8epZusdV5tncsUSVwT4In0mRRGB0fhc0z4aeHrbXb5SsXu8lhbaP4ddMh5qQcpE+zMAcE\nqVQpZAzsmGcNN+1ZChVCodtLEH8H+AcVudm9x9N56acU5qYcolbVCnxxe2u6NKjmwMDLHk0WheHj\nB/0/hHFXwqzHYcCnxW6yU71QIiqV4+vluzVZqLLHGGs+cOHrsH8VBIVBz/9Ci5utWm1FdC4rlw9/\n38FHv+/Ax0t4okdD7rgixuNXKpUETRaFVaMpdHocFrxirY5q1LdYzXl5CUMSovjvL1vYfvgMdasF\nOihQpdxYXq41Yb3oTTi0AUKioe870HywNUdYRMYYZm88yMs/bWJf2jmubR7GU70aUaOiVkpwFB0s\nvxwdH4EazeCnkXD2WLGbuzE+El9vYULibgcEp5Qby82BNRPh/QSYcjvkZEL/j+GBVdDqtmIliu2H\nz3DL5yu45+tVBPr7MGl4W94d3EIThYNpz+JyePtaK6I+7gwzH4WBXxSruaqB/vRoUpPvk1N5vHtD\nXZ2hPE9OpnUT3eIxkLYbqjeBgeOtxSJexft5P52RzXvztvP54j8o5+fNC31jGdY2Gh9v/QzsDJos\nLlf1xnDlkzDvZat2VOP+xWpuaEIUP67dz0/r9jMwPtJBQSrlYlnpsOpLWPIOnN4PYS2h52tQv0ex\nK8AaY/hhzT5embmZI6czuSk+ksd6NKBqYNF7J8o+pyYLEekBvAN4A58aY0Zf8PoY4PzGEeWBasaY\nENtrUcCnQCRggF7GmF3OjLfQOjxs7eH7878h+goIDC1yUwm1KlO3WiBfJ+7RZKFKv8zTVmG/ZWPh\n7BGIam+Vy6ndxSFlwjfuP8kLMzayctcJmkdU5JNb4omL1F3uSoLTkoWIeAPvA92AVGCliMwwxqSc\nP8YYMzLf8Q8A+ctFfgmMMsbMFZFA4O8lHF3J2weu+wg+7gg/PwI3flnkXwQRYWhCFC/+mMKGfSdp\nEl7RwcEqVQLOnYDEcdZeEhlpUOcqa8l5TAeHNJ+WnsWbc7YyIXE3IeX9eG1AUwa2iizzJThKkjMH\n99oA240xO40xWcAkoN8ljh8MTAQQkVjAxxgzF8AYc8YYk+7EWC9ftYbQ5SnYNAM2Ti1WU9e3jCDA\n14sJiXscFJxSJeTsUfj1RRjT1FopGNUO7poHN09zSKLIzTN8k7iHLm8sYELibm5pF8P8f1/JTa2j\nNFGUMGcOQ4UDe/M9TgUSCjpQRKKBWsA821P1gTQRmWp7/lfgSWNM7gXnDQeGA0RFRTk0+EJp9wBs\nyjccFVS0woAVy/lybfMwpq/Zx1O9GhKkdWqUuzt1wNq2NPkLyD4Hja+zyoTXaOqwS6zac4Lnp29k\n/b6TtKlVmRevbUyjmo7bvVJdHmf2LApK+6aA5wAGAVPyJQMfoCPwKNAaqA3c9o/GjBlnjIk3xsSH\nhhZ93qDIvH2s2lFZ6dZy2gsL3F+GoQnRpGfl8sPqfQ4MUCkHS9sDPz0C7zSDxI+sVU0jVlgrnByU\nKI6czuTR79Zy/QdLOXw6g3cGxfHt8LaaKFzssnsWIuIFBBpjTtk5NBVrcvq8CGD/RY4dBIy44NzV\nxpidtmv+ALQFPrvceJ0utD5c/ay1Dev676DZjUVqpllERZqEBzMhcQ/D2kYjbrhnsCrDjIGVn1oF\nNQFaDLUWelSu5bBLZOfm8eWy3bw9dysZObnc07kO919Vl0B/XbTpDgrVsxCRb0QkWEQqACnAFhF5\nzM5pK4F6IlJLRPywEsKMAtpuAFQCll1wbiUROd9duMp2XffU9j6ITICZj1nd8yIQEYYlRLP54GmS\nd59wcIBKFUNWOvxwr3VvUZ0u8NAa665rByaKpTuO0vvdRbz8Uwotoivxy8OdeLJnQ00UbqSww1Cx\ntp7EdcBMIAq4+VInGGNygPuB2cAmYLIxZqOIvCQi1+Y7dDAwyZi/xnBsw1GPAr+JyHqsIa1PChlr\nyfPyhn4fQE6GVWywiMNRfZuHEeTvoxPdyn0c/wM+uwbWTrK2Gx78LVSMcFjz+9POMeKbVQz5JJH0\nrFzG3dyK/93emjqhWv7G3RQ2bfuKiC9WshhrjMkWEbt/EY0xM7GSS/7nnrvg8QsXOXcu0KyQ8ble\n1bpw9fMw+z/WL1bc4MtuooK/D/1bhjNp5V6e7RNL5Qp+TghUqULaNhe+vwswMGSyw3aLBMjMyeXT\nRX8wdt528oxhZNf6/KtzbQJ8tYqBuypsz+JjYBdQAVhoW71kb86i7Em4x7oJadYTcOpi0zOXNjQh\nmqycPKYk77V/sFLOkJcHC0bDhIFQMRKG/+7QRDFv8yG6j1nI67O30Ll+KL8+0pmHutbTROHmCpUs\njDHvGmPCjTG9jGU3f915rc7z8oJ+YyEvG2Y8WKThqAY1gmgdU4lvEveQl1f01VVKFcm5EzBxECx4\nFZrdBHfOcdjcxK6jZ7lz/EruGJ+El5fw5R1t+OjmVrq1cClR2Anuh2wT3CIin4nIKqxJZ3WhKnWg\n64uwfS6s/rpITQxrG82uY+ks2XHUwcEpdQkH11t7tuyYB73esIpmFmNvifPSs3J4Y/YWrhmzkOU7\nj/FUr4b88lAnOtV3wXJ3VWSFHYa6wzbBfQ0QCtwOjL70KWVY67sgpiPMfgrSLn84qUeTGlSu4MeE\n5TrRrUrI2m/h025WldjbZ1rbmTqg4N/P6w7Q9c3fGTt/O72b1WTeo1cyvFMd3Uq4FCrs/7HzPzW9\ngC+MMWsp+KY7BdZw1LXvWRu9zHjgsoej/H28GdgqgrmbDnHoVIaTglQKyMmylnxPGw7hLa35icg2\nxW5266HTDPkkkRHfrKJieT8m/6sdY26Ko3qw7jFRWhU2WSSLyBysZDFbRIJwp8J+7qhyLbjmJWvz\n+VX/u+zThyREkZtnmLRCJ7qVk5w6AP/rAyvGQbv74ZbpRS5Z82eTGdm8/FMKPd9ZRMqBU7zcrzE/\n3t+BNrWKv2+9cq3CLp29E4gDdhpj0kWkCtZQlLqUVndAygyY/bRVhTOk8PWroqtUoGO9qkxauYcR\nXerohi7KsXYtge9ug6yzcMPn0GRAsZrLyzNMXb2P0bM2c+xsJoNaR/FY9wa6/NuDFHY1VB5WuY5n\nROQNoL0xZp1TI/ME51dHAUwfYS1JvAxDE6I5cDKD+VuOOCE4VSYZA8s+gP/1hYBguPu3YieK9akn\nGfDRUh79bi0RlcoxfUQHXr2+qSYKD1PY1VCjgYewSm6kAA+KyKvODMxjhERB91Hwx0JI/vyyTu3a\nqBrVg/35ernu0a0cIPMMfH+ndeNog55w9zyo1qjIzZ04m8VT09Zz7fuL2Xs8nddvaMbUe9vTLEI3\nI/JEhR2G6gXE2XoYiMj/gNXAf5wVmEdpeSukTIc5z0HdrlApplCn+Xh7Mah1FO/O28aeY+lEVdH1\n6KqIju2ASUPh6Bar0kCHh62ebxHk5hm+WbGHN2Zv4UxmDre3r8XD3eoRrKX1Pdrl/LTk/7ig27ld\nDhFrdZSXN0y//7KGowa1icRLhIkrdRmtKqLNP1v3T5w5BMOmQsdHipwoknYdp+97i3n2hw3E1gxm\n5oMdea5vrCaKMqCwPzGvAqtFZLytV5EMvOK8sDxQxQjo/grsWmSVei6kmhXLcXXDakxeuZfMnFz7\nJyh1Xl4u/PYSTBpi3Sz6r9+tqrFFNHvjQQZ+vIwT6VmMHdKCb+5OoEGNIAcGrNxZYSe4J2LtJzHV\n9tXOGDPJmYF5pBbDoG43+PV5a1igkIa2jebY2SxmbzzkxOCUR0k/DhNugEVvQstb4PZfLms13oW2\nHTrNI9+uoVl4RX59pDN9moXpnitlzCWThYi0PP8F1MTalGgvEGZ7Tl0OEbj2XfDyvazVUR3rViWq\ncnkm6ES3Koz9q+HjzrBrMfR91xoC9S36zXAnz2Uz/Ktkyvl589HNraige0yUSfb+r795idcMWh/q\n8gWHQc/R1mYyiR9Bu/vsnuLlJQxJiGL0rM1sO3SaetW1668uYtVX1p7wFULhjl8gvFWxmsvNMzw0\naTV7j6czcXhbalYs56BAVWlzyZ6FMabLJb40URRV88FQvwf89iIc3V6oUwa2isDXW3RjJFWwnEz4\n8SGYcT9EtbXmJ4qZKADemruFBVuO8MK1jWkdo3dhl2WFvc/i+gK+rhaRas4O0COJQJ+3wScApt9n\nTUTaUSXQn55NavL9qlTOZelEt8rnZCp83gOSx8MVI+HmaVCharGb/XndAd6fv4NBrSMZmlD0+Q7l\nGQq7GupO4FNgqO3rE+ARYImIXHJ7VXURwTWh1+uwNxGWf1CoU4a1jeZ0Rg4/ri3axkrKA+1cAB93\ngqPb4KavoesL1hLtYtp88BSPfreWFlEhvNivsU5mq0InizygkTFmgDFmABALZAIJwBPOCs7jNR0I\nDfvAby/Dka12D28dU4l61QKZkKgT3WWeMbB4DHzV35qfGD4fGvV1SNNp6VkM/zKZoAAfPhrWCn8f\n3cFOFT5ZxBhj8q/bPAzUN8YcB7IdH1YZIQJ9xoBfBfjhHsjNsXO4MDQhirWpJ1mferKEglRuJ+MU\nTL4Zfn0BGl0Ld/0GVes5pOmc3DwemLiagycz+OjmVlpSXP2psMlikYj8JCK3isitwAysvbgrAGnO\nC68MCKxmDUftS4Zl79k9/PqXAYdZAAAfmklEQVRWEZTz9dbeRVl1ZAt8chVsngnXjIKB48E/0GHN\nvz57C4u2HeWlfo1pGVXJYe2q0q+wyWIE8AVWmfIWwP+AEcaYs8YY3Yu7uJoMsD4hzn8FDm+65KHB\nAb5c2zyM6Wv2cypDO3VlysZpVqLISLP2nmh/f7F3s8tv+pp9fLxwJ8PaRjGojU5oq78r7B3cBlgM\nzAN+BRbanlOOIAK93wL/IOv+CzvDUcPaRnMuO5dpq/aVUIDKpXJzYM4z1v4T1RpZu9nV6ujQS2zc\nf5Invl9H65hKPNensUPbVp6hsEtnbwRWADcANwKJInJDIc7rISJbRGS7iDxZwOtjRGSN7WuriKRd\n8HqwiOwTkbGFezulWGAo9H7Tuvt2yduXPLRpREWaRVRkQuJuNGd7uDNH4KvrYOl71t7ut82EiuEO\nvcTxs9aEdkg5Pz4Y2kr3x1YFKuxPxdNAa2PMrcaYW4A2wLOXOkFEvIH3gZ5Yq6cGi0hs/mOMMSON\nMXHGmDjgPay6U/m9DPxeyBhLv8b9ofH1sGA0HNp4yUOHJkSx9dAZknafKKHgVInbu9JaFpu6Eq77\nyPow4ePYDYVycvMYMWEVR85k8vHNrQgN8ndo+8pzFDZZeBljDud7fKwQ57YBthtjdhpjsoBJQL9L\nHD8YmHj+gYi0AqoDcwoZo2fo9QaUC4Fp90Duxeck+jYPIyjARzdG8kTGwMrP4Iue4O0Ld86FuMFO\nudQrMzezbOcxRl3XhOaRummRurjCJotfRGS2iNwmIrcBPwMz7ZwTjlV08LxU23P/ICLRQC2sORFE\nxAurLtVjhYzPc1SoYi2nPbgOFr110cPK+/kwoGUEs9Yf5NiZzBIMUDlV9jn44T74+RGofSUMXwA1\nmznlUlNXpfL5kj+4rX0MA+MjnXIN5TkKO8H9GDAOaAY0B8YZY+zdjFfQMo2LDbAPAqYYY87XsbgP\nmGmM2XuR460LiAwXkSQRSTpyxIP2qW7U17phb+F/4cDFtzofkhBFVm4eU5JTSzA45TQndsFn18Da\nb6DzEzBkMpR3Tj2mdalpPDl1PW1rV+bp3kXfWlWVHYWeyTLGfG+MecQ2zzCtEKekAvk/rkQAF6tT\nMYh8Q1BAO+B+EdkFvAHcYtsH/MKYxhlj4o0x8aGhoYV6H6VGz/9C+SrWp8ycrAIPqV89iDa1KvPN\nij3k5elEd6m27VerrHjabhj8LXR5qsi72dlz5HQm//oqmdBAf94f0hJfb53QVvbZ28/itIicKuDr\ntIicstP2SqCeiNQSET+shDCjgGs0ACoBy84/Z4wZaoyJMsbEAI8CXxpj/rGayqOVr2wVGzy0Hha9\ncdHDhiZEsftYOou3Hy3B4JTD5OXB7/+1NiqqGGENOzXo4bTLZefmMeKbVRw/m8XHN7eiSqBOaKvC\nsVeiPMgYE1zAV5AxJtjOuTnA/cBsYBMw2RizUUReEpFr8x06GJik920UoGEvq5z5wjdg/5oCD+nR\npAZVKvjpHd2l0bk0mDQY5o+CZjdaE9mVazv1kv/3Uwor/jjOawOa0SS8olOvpTyLeMrf6Pj4eJOU\nlOTqMBzv3An4oB2Uq2R96vT55yfB0bM288minSx54ipqVNRaPqXCwQ3w7TA4uRe6vwpt7nbo3dgF\nmbxyL49/v467O9bi6d6x9k9QZYKIJBtj4u0dp4OV7q5cJej7DhxOgd9fK/CQIW2iyM0zTFqpGyOV\nCuu+g0+7WiufbvsZEoY7PVGs3nOCZ37YQIe6VXiiR0OnXkt5Jk0WpUH97hA3zCpJvS/5Hy9HVSlP\np/qhTFqxl5zcwu3rrVwgNxtmPQFT74KwFvCvhdaudk52+HQG93ydTLVgf8YObomPTmirItCfmtKi\n+ygIqmmtjsrO+MfLwxKiOHgqg982Hy7gZOVypw/C+D7Wvutt74NbZ0BQdadfNisnj3u/XsWpczmM\nuzmeShUcewe4Kjs0WZQW5ULg2nfhyGZY8Oo/Xr6qYTVqBAfoHt3uaNdiq2zHwXUw4DPo8ap1Z3YJ\neOHHjSTvPsHrA5sRG3bJNSlKXZImi9KkbldoeQssfdeqG5SPj7cXg9pEsnDrEdal6hYjbmH/Gvhm\nEIzvDX6B1iZFTe3W33SYbxL38E3iHu7pXIc+zcJK7LrKM2myKG2uGQVBYVYp8+xzf3vp9va1qBEc\nwMhv15CRnXuRBpTTHVgHE4fAuM6wZyl0eQb+9TtUL7kVSMm7j/P8jA10qh/KY90blNh1lefSZFHa\nBARDv7FwbBvM+7+/vVSxvC+vD2zGjiNnee2XzS4KsAw7vxz2447W0NOVT8HD66HzY9ZeJSUVxskM\n7vl6FWEh5XhvUAu8vZy70kqVDT6uDkAVQZ0uEH8HLHvfqiOVb0VNx3qh3Noumi+W7KJro+p0qFvV\nhYGWEYdS4PfRkDId/IOh85PQ9l5rnqmEZWTn8q+vkzmbmcOEuxKoWL5k5kaU59OeRWnV7SUIibRW\nR2Wl/+2lJ3s2onZoBR79bi0nz+nWq05zeLO1e92H7WH7POj0GDy8Drr8xyWJwhjDc9M3sHZvGm/d\n2Jz61UuuN6M8nyaL0so/CPq9D8d3wLyX//ZSOT9vxtwYx+HTmTw/fYOLAvRgR7bClDvhg7awbS50\nfMRKElc9Y91E6SJfL9/N5KRUHriqLj2a1HRZHMoz6TBUaVarE7S+G5Z/CA37QEyHP19qHhnCA1fV\n5e1ft9E1trquhnGEo9utu+g3TAGfcnDFw9DuAWsPEhdL3HmMF39M4eqG1RjZtb6rw1EeSHsWpV3X\nF6BSNEy/D7LO/u2lEV3q0jwyhKenbeDQqX/eyKcK6dgOa+fC91vD5p+g3f1WT6LrC26RKPanneO+\nCauIqlKeMYPi8NIJbeUEmixKO/9A6PeBtXHOry/87SVfby/G3NiczJxcHpuyDk8pGlliju+05oTG\ntoaNP1h3Xj+0Dq55GSq4x8KBjOxc/vVVMpk5eYy7OZ7gAJ3QVs6hycITxHSAhHthxTj4Y+HfXqod\nGsjTvRqxcOsRvta7uwvnxC6YPgLei4cN30PCPfDQWqvkSqD7bLJljOGpaetZv+8kY26Ko261QFeH\npDyYzll4iqufg22zYdq90PlxqHcNBFuTnMPaRjN302FG/ZxChzpVqB2qf1QKlLbH2jtkzQQQb6ts\n+BUjIaiGqyMr0BdLdjF11T5Gdq1Pt1jn15lSZZvuZ+FJUpOtpZwnbT2ImnFWxdr63TkU2Ihr3l5M\nTNUKfH9PO608ml/aXlj0Jqz+2ioV3uo2K0kEu++igKXbj3Lz5yu4umE1PhrWSucpVJEVdj8LTRae\nxhhr74utv8DW2bB3BWCgQjX2VL2CUduiiOvcn3u7x7k6Utc7uc9KEqu+tB63uhWueAQqhrs2Ljv2\nHk/n2rGLqRLozw8jOhDorwMEqug0WSjL2WOw/VcreWz/DTJPkmW8yQxvR1CzPlbPw8lbebqdUwdg\n8VuQPN5Kri2GQcd/Wzc5urlzWbkM+HApe0+kM31EBx1SVMWmyUL9U242Z7Yt4YfJn9ORZKLzUq3n\nq9a35jjq97BKh5RQ+ewSd/qgtYFU0hdgciFuKHR6FEKiXB1ZoRhjeGjSGn5ct5/Pb21Nl4bVXB2S\n8gCFTRbafy1LvH0JbHglMYOb0PmzREa28uGhyD+sifEV42DZWPCvCHWvshJH3W5ucR9BsZ0+BEve\nhqTPrd3q4oZYSaJSjKsjuyyfLNrJjLX7eax7A00UqsRpsiiDrqhXldvaxzBm6S5axg2gY9t7IPM0\n7FxgzXNsmwMbpwECEa1tk+Q9oHpjp+8V7VBnjlhJYuVnkJsFzQdZSaIUDrst2naE0bM206tpDe67\nso6rw1FlkA5DlVEZ2bn0fncRZzNzmf1wp79XJ83LgwNrrKSx9RfYv9p6PjgC6tuGq2p1At9yrgne\nnrNHYck7sPJTyMmAZjdZRf6qlM4/snuOpdN37GJqBAcw9b72VNAJbeVAOmeh7FqXmsb1Hyyld7Oa\nvDOoxcUPPH3QKpi39RfYMR+yz1q1kWp1+nNpLhUjSi7wizl7zNpFcMUnkHMOmtxg3XNStZ6rIyuy\ns5k5DPhwKQdOZjDj/g5EV6ng6pCUh9E5C2VXs4gQHry6Hm/N3UrXRtXp2/wi9xUE1YCWN1tfOZmw\ne4k1XLX1F2u+42egepO/hqvCW4GXd8m9kfTj1nxL4sdWfawmA6DzExBaugvqGWN4bMpath46zfjb\n22iiUC7l1J6FiPQA3gG8gU+NMaMveH0M0MX2sDxQzRgTIiJxwIdAMJALjDLGfHupa2nPomhycvO4\n4aNl/HH0LLMf7kSNigGFP9kYOLrtr3s69iyzVhmVr2JNjtfvDnWuct7eDudOWBtALf8Iss5A4+us\njYeqNXTO9UrY+/O38/rsLTzVqyHDO5XOITTl/lw+DCUi3sBWoBuQCqwEBhtjUi5y/ANAC2PMHSJS\nHzDGmG0iEgYkA42MMWkXu54mi6LbeeQMvd9dTHxMJb68ow1S1Ensc2mw4zfbJPlcOHfcKpsR3d5K\nHPW6W0NCxZ0kP5dmlWVf/gFknoLYflaSKME9rp1t/pbD3DF+JX2bhfHOoLii/z9Ryg53GIZqA2w3\nxuy0BTQJ6AcUmCyAwcDzAMaYreefNMbsF5HDQChw0WShiq52aCBP9W7Esz9s4Kvlu7mlXUzRGioX\nYg0BNRkAebmQmmQbqpoDc56xvirVsoaq6l8D0R3Ax7/w7WectHoRy96HzJPWlrKdn4QaTYoWr5v6\n4+hZHpy4mkY1gnltQDNNFMotODNZhAN78z1OBRIKOlBEooFawLwCXmsD+AE7CnhtODAcICqqdNxY\n5a6GJUTxa8ohXpm5iQ51q1KnuHcGe3lDVIL11fV5q/7SttmwdQ4kfwGJH4JfoLWfeL3u1k2BQRcp\nhpdxypqPWDYWMtKgQW+48kmo2ax4MbqhM5k53P1lEj5ewsc3t6KcXwnO/Sh1Cc4chhoIdDfG3GV7\nfDPQxhjzQAHHPgFEXPiaiNQEFgC3GmOWX+p6OgxVfIdOZdD97YVEVy7PlHvb4+usYoNZ6VYp9W2z\nrSGrU/us58Na/rW6qkZza9XVinGw9D1rfqJ+TytJhHlmXau8PMM9Xyfz2+bDfHVHG9rXdY89M5Rn\nc4dhqFQgf7GdCGD/RY4dBIzI/4SIBGOts3nGXqJQjlE9OIBR1zVlxDereH/+dh521vacfuWhQQ/r\nyxg4tPGvSfIFo2HBqxBYw7qR7txxq+dx5ZMQ3tI58biJsfO3MyflEM/2idVEodyOM5PFSqCeiNQC\n9mElhCEXHiQiDYBKwLJ8z/kB04AvjTHfOTFGdYHezWoyNyWM9+Ztp0uDajSPdNJKpvNErDmHGk2s\nu6vPHv2r8GFeLnR4GCJaOTcGNzA35RBvzd3K9S3CuaNDjKvDUeofnL10thfwNtbS2c+NMaNE5CUg\nyRgzw3bMC0CAMebJfOcNA74ANuZr7jZjzJqLXUuHoRzn5Llsery9kHK+3vz8YEcdN3ey7YfPcN37\nS6hVtQLf3dOOAF/991Ylx+VLZ0uaJgvHWrL9KEM/TeTWdtG82M+zVhu5k1MZ2Vw3dgknz2Xz4wNX\nEBbipiVUlMcqbLLQ7dJUgTrUrcrtHWL437LdLNx6xNXheKS8PMPISWvYczydD4a21ESh3JomC3VR\nT/RoSN1qgTw2ZS1p6VmuDsfjvP3rVn7bfJjn+saSUNsDSsErj6bJQl1UgK83b98Ux7EzWTw7faP9\nE1Sh/bLhAO/O286N8RHc3Dba1eEoZZcmC3VJTcIr8tDV9fhx7X6mr9nn6nA8wtZDp/n35LXERYbw\nUr8meoe2KhU0WSi77r2yDi2iQnj2hw0cOHnO1eGUaifTsxn+ZRLl/X34aFgrXfmkSg1NFsouH28v\nxtwYR3au4fEp68jL84wVdCUtN8/w4KTV7Es7x4dDW15ehV+lXEyThSqUmKoVeLp3IxZtO8pXy3e7\nOpxS6Y05W/h96xFevLYJ8TGVXR2OUpdFk4UqtKEJUVzZIJRXZm5i++Ezrg6nVPlp3X4+XLCDIQlR\nDEnQopeq9NFkoQpNRPjvgGaU9/PmkclryM7Nc3VIpcKmA6d47Lt1tIquxAt9G7s6HKWKRJOFuizV\nggMY1b8p61JPMnbedleH4/YWbzvKrZ+vILicDx8ObYmfj/7KqdJJf3LVZevVtCbXtwhn7PztrNmr\n+1EVJCM7lxdmbGTYZ4kEBvgw/vY2VAvWCW1VemmyUEXyQr/GVA/y55Fv13AuK9fV4biV9akn6fPe\nYsYv3cVt7WP4+YGONKoZ7OqwlCoWTRaqSIIDfHnjxubsPHqWV2dtcnU4biEnN4+x87bR/4MlnM7I\n5qs72/DCtY21aq/yCM7cz0J5uPZ1qnLnFbX4bPEfXN2oOp3rh7o6JJfZdfQsIyevYfWeNPo2D+Pl\nfo0JKe/n6rCUchjtWahieax7A+pVC+Sx78pmsUFjDBMSd9PznUXsOHyGdwbF8d7gFpoolMfRZKGK\nJcDXmzE3xXEiPYunf9iAp+yPUhiHT2dwx/iVPD1tA62iKzF7ZCf6xYW7OiylnEKThSq2JuEVebhr\nfX5ed4AZay+2zbpn+WXDAbqPWcjSHcd4oW8sX97RhpoVdT8K5bl0zkI5xL861ea3TYd49ocNtKlV\n2WP/cJ7KyObFGSl8vyqVpuEVGXNTc+pWC3J1WEo5nfYslEP4eHvx1o1x5OQZHv1urUcWG1y+8xg9\n317EtNWpPHhVXabe114ThSozNFkoh4mpWoFneseyZPsx/rdsl6vDcZiM7FxembmJwZ8sx9dbmHJv\nex65pgG+3vrro8oOHYZSDjW4TSS/bjrE6Fmb6Vivaqn/5J2y/xQjv13DlkOnGZoQxdO9G1HeT39t\nVNmjH42UQ4kIowc0pbyfNyO/XVtqiw3m5hk+XLCDfu8v5nh6Fl/c1ppR/ZtqolBlliYL5XDVggJ4\n9fqmrN93kvd+2+bqcC7b3uPpDBq3jNd+2UzXRtWZ/XAnujSs5uqwlHIppyYLEekhIltEZLuIPFnA\n62NEZI3ta6uIpOV77VYR2Wb7utWZcSrH69GkJte3DOf9BTtYveeEq8MpFGMMk1fupcfbC9l84DRv\n3dicD4a2pHIFvcFOKXHWTVQi4g1sBboBqcBKYLAxJuUixz8AtDDG3CEilYEkIB4wQDLQyhhz0b86\n8fHxJikpycHvQhXHqYxser69CD8fL35+8Aq3HsI5eiaT/0xdz9yUQ7StXZk3BjYnolJ5V4ellNOJ\nSLIxJt7ecc7sWbQBthtjdhpjsoBJQL9LHD8YmGj7vjsw1xhz3JYg5gI9nBircoLgAF/eGNicXcfO\n8spM9y02ODflED3eXsjvW47wTO9GfHNXW00USl3AmR/1woG9+R6nAgkFHSgi0UAtYN4lztU6CqVQ\nuzpVuLNDLT61FRvs0sB9xv7PZObwfz+lMGnlXhrVDGbCXXE0qFG6V28p5SzO7FlIAc9dbMxrEDDF\nGHN+Y4RCnSsiw0UkSUSSjhw5UsQwlbM92r0B9asH8viUdZw46x7FBpN2HafnOwv5Nmkv915Zhx9G\ntNdEodQlODNZpAKR+R5HABcrHDSIv4agCn2uMWacMSbeGBMfGlp2y2O7u/PFBtPSs3jGxcUGs3Ly\neO2Xzdz48TIAJv+rHU/0aIi/j+45odSlODNZrATqiUgtEfHDSggzLjxIRBoAlYBl+Z6eDVwjIpVE\npBJwje05VUo1DrMVG1x/gOlrXFNscMvB01z3/hI+XLCDga0imfVQJ1rHVHZJLEqVNk6bszDG5IjI\n/Vh/5L2Bz40xG0XkJSDJGHM+cQwGJpl8HzeNMcdF5GWshAPwkjHmuLNiVSXjns51mLf5MM9Ot4oN\nhoWUTLHBvDzD50v+4L+ztxDk78Mnt8TTLbZ6iVxbKU/htKWzJU2XzpYOu4+dpec7i4iLDOHrOxPw\n8ipoespx9qWd49HJa1m28xhdG1Vn9ICmVA30d+o1lSpN3GHprFL/EF2lAs/2iWXpjmOMX7rLadcx\nxjB1VSo9xixkXWoa/x3QjE9uaaWJQqkict+7pJTHGtQ6kl9TDjH6F6vYYL3qjl2FdOJsFk//sJ6Z\n6w/SOqYSbw6MI6qK3jehVHFoz0KVOKvYYDMC/X0YOXkNWTmOKzY4f8thrnl7IXNTDvFEj4ZMGt5O\nE4VSDqDJQrlEaJA/r/RvyoZ9p3hvXvGLDaZn5fD0tPXc/sVKKpf344cRHbj3yjp4O3lORKmyQoeh\nlMv0aFKDG1pF8P787VzZoBqtoisVqZ3Ve07wyOS17Dp2lrs71uLf1zQgwFfvm1DKkbRnoVzq+b6x\n1KxYjn9PXkN6Vs5lnZudm8dbc7Zww0fLyMrJ45u72vJ071hNFEo5gSYL5VJBAb68eWNzdh9PZ9TP\nhS82uP3wGa7/YCnvzttOv7gwZj3ckXZ1qjgxUqXKNh2GUi7XtnYV7u5Ym3ELd9K1UfVLbjSUl2f4\nctkuXp21mfJ+3nw4tCU9m9YsuWCVKqO0Z6HcwiPd6tOgehCPf7+O4xcpNnjwZAa3frGCF35MoX2d\nKsx+uJMmCqVKiCYL5RbyFxt8etr6fxQbnLF2P9eM+Z2kXScY1b8Jn9/WmmrBAS6KVqmyR5OFchux\nYcE80q0BszYcZNrqfQCcTM/mgYmreXDiaupUC2TmQx0ZmhCNiC6JVaok6ZyFcivDO9Vm3uZDPD99\nI8bA67O3cPRMJv/uVp97r6yDj7d+vlHKFfQ3T7kVby/hzYFx5BnDv79bSwV/b6bd14EHrq6niUIp\nF9KehXI7UVXKM3ZIS9ampnFP5zp634RSbkCThXJLXRpWu+QSWqVUydJ+vVJKKbs0WSillLJLk4VS\nSim7NFkopZSyS5OFUkopuzRZKKWUskuThVJKKbs0WSillLJLLqzuWVqJyBFgdzGaqAocdVA4ruQp\n7wP0vbgrT3kvnvI+oHjvJdoYE2rvII9JFsUlIknGmHhXx1FcnvI+QN+Lu/KU9+Ip7wNK5r3oMJRS\nSim7NFkopZSyS5PFX8a5OgAH8ZT3Afpe3JWnvBdPeR9QAu9F5yyUUkrZpT0LpZRSdmmysBGRl0Vk\nnYisEZE5IhLm6piKSkReF5HNtvczTURCXB1TUYnIQBHZKCJ5IlLqVq6ISA8R2SIi20XkSVfHUxwi\n8rmIHBaRDa6OpThEJFJE5ovIJtvP1kOujqmoRCRARFaIyFrbe3nRadfSYSiLiAQbY07Zvn8QiDXG\n3OPisIpERK4B5hljckTkNQBjzBMuDqtIRKQRkAd8DDxqjElycUiFJiLewFagG5AKrAQGG2NSXBpY\nEYlIJ+AM8KUxpomr4ykqEakJ1DTGrBKRICAZuK40/n8REQEqGGPOiIgvsBh4yBiz3NHX0p6FzflE\nYVMBKLVZ1BgzxxiTY3u4HIhwZTzFYYzZZIzZ4uo4iqgNsN0Ys9MYkwVMAvq5OKYiM8YsBI67Oo7i\nMsYcMMassn1/GtgEhLs2qqIxljO2h762L6f87dJkkY+IjBKRvcBQ4DlXx+MgdwCzXB1EGRUO7M33\nOJVS+kfJU4lIDNACSHRtJEUnIt4isgY4DMw1xjjlvZSpZCEiv4rIhgK++gEYY542xkQCE4D7XRvt\npdl7L7ZjngZysN6P2yrMeymlpIDnSm2P1dOISCDwPfDwBSMLpYoxJtcYE4c1gtBGRJwyROjjjEbd\nlTGmayEP/Qb4GXjeieEUi733IiK3An2Aq42bT0xdxv+X0iYViMz3OALY76JYVD628f3vgQnGmKmu\njscRjDFpIrIA6AE4fBFCmepZXIqI1Mv38Fpgs6tiKS4R6QE8AVxrjEl3dTxl2EqgnojUEhE/YBAw\nw8UxlXm2SeHPgE3GmLdcHU9xiEjo+dWOIlIO6IqT/nbpaigbEfkeaIC18mY3cI8xZp9royoaEdkO\n+APHbE8tL8Uru/oD7wGhQBqwxhjT3bVRFZ6I9ALeBryBz40xo1wcUpGJyETgSqwKp4eA540xn7k0\nqCIQkSuARcB6rN93gKeMMTNdF1XRiEgz4H9YP19ewGRjzEtOuZYmC6WUUvboMJRSSim7NFkopZSy\nS5OFUkopuzRZKKWUskuThVJKKbs0WSh1GUTkjP2jLnn+FBGpbfs+UEQ+FpEdtoqhC0UkQUT8bN+X\nqZtmlXvTZKFUCRGRxoC3MWan7alPsQrz1TPGNAZuA6raig7+BtzkkkCVKoAmC6WKQCyv22pYrReR\nm2zPe4nIB7aewk8iMlNEbrCdNhSYbjuuDpAAPGOMyQOwVaf92XbsD7bjlXIL2s1VqmiuB+KA5lh3\nNK8UkYVAByAGaApUwyp//bntnA7ARNv3jbHuRs+9SPsbgNZOiVypItCehVJFcwUw0Vbx8xDwO9Yf\n9yuA74wxecaYg8D8fOfUBI4UpnFbEsmybc6jlMtpslCqaAoqP36p5wHOAQG27zcCzUXkUr+D/kBG\nEWJTyuE0WShVNAuBm2wbz4QCnYAVWNtaDrDNXVTHKrx33iagLoAxZgeQBLxoq4KKiNQ7v4eHiFQB\njhhjskvqDSl1KZoslCqaacA6YC0wD3jcNuz0PdY+Fhuw9g1PBE7azvmZvyePu4AawHYRWQ98wl/7\nXXQBSl0VVOW5tOqsUg4mIoHGmDO23sEKoIMx5qBtv4H5tscXm9g+38ZU4D+leP9x5WF0NZRSjveT\nbUMaP+BlW48DY8w5EXkeax/uPRc72bZR0g+aKJQ70Z6FUkopu3TOQimllF2aLJRSStmlyUIppZRd\nmiyUUkrZpclCKaWUXZoslFJK2fX/YOY2L9CAyrgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x110437f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "logloss_s1 =np.array(logloss_s).reshape(len(Cs),len(penaltys))\n",
    "x_axis = np.log10(Cs)\n",
    "for j, onePenalty in enumerate(penaltys):\n",
    "    pyplot.plot(x_axis, np.array(logloss_s1[:,j]), label = ' Test-' + onePenalty)\n",
    "\n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'log(C)' )                                                                                                      \n",
    "pyplot.ylabel( 'logloss' )\n",
    "\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 用最佳参数在全体训练数据上训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bestCs = logloss_s1.argmin(axis = 0)\n",
    "\n",
    "best_logloss = logloss_s1[bestCs[0],0]\n",
    "best_penalty_index = 0\n",
    "best_penalty = penaltys[best_penalty_index]\n",
    "\n",
    "for j, onePenalty in enumerate(penaltys):\n",
    "    if logloss_s1[bestCs[j],j] < best_logloss:\n",
    "        best_logloss = logloss_s1[bestCs[j],j]\n",
    "        best_penalty_index = j\n",
    "        best_penalty = penaltys[best_penalty_index]\n",
    "       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best C: 0.100000 \n",
      " best penalty: l1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=0.1, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l1', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bestCs = logloss_s1.argmin(axis = 0)\n",
    "\n",
    "best_logloss = logloss_s1[bestCs[0],0]\n",
    "best_penalty_index = 0\n",
    "best_penalty = penaltys[best_penalty_index]\n",
    "\n",
    "for j, onePenalty in enumerate(penaltys):\n",
    "    if logloss_s1[bestCs[j],j] < best_logloss:\n",
    "        best_logloss = logloss_s1[bestCs[j],j]\n",
    "        best_penalty_index = j\n",
    "        best_penalty = penaltys[best_penalty_index]\n",
    "\n",
    "bestC = Cs[bestCs[best_penalty_index]]\n",
    "\n",
    "print(\"best C: %f \\n best penalty: %s\"%(bestC, best_penalty) )\n",
    "    \n",
    "LR = LogisticRegression(penalty=best_penalty, C=bestC)\n",
    "LR.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 若计算资源允许，可用下面的GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'penalty': ['l1', 'l2'], 'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "#需要调优的参数\n",
    "# 请尝试将L1正则和L2正则分开，并配合合适的优化求解算法（slover）\n",
    "#tuned_parameters = {'penalty':['l1','l2'],\n",
    "#                   'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "#                   }\n",
    "penaltys = ['l1','l2']\n",
    "Cs = [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "tuned_parameters = dict(penalty = penaltys, C = Cs)\n",
    "\n",
    "lr_penalty= LogisticRegression()\n",
    "grid= GridSearchCV(lr_penalty, tuned_parameters,cv=5, scoring='neg_log_loss')\n",
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "F:\\Soft\\AI\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "F:\\Soft\\AI\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "F:\\Soft\\AI\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "F:\\Soft\\AI\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "F:\\Soft\\AI\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "F:\\Soft\\AI\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "F:\\Soft\\AI\\Anaconda2\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([  1.18700004e+00,   6.24260001e+00,   1.24214000e+01,\n",
       "          1.33118001e+01,   3.85945999e+01,   4.17720000e+01,\n",
       "          1.62263600e+02,   1.06797400e+02,   1.93979960e+03,\n",
       "          2.08789400e+02,   2.07448680e+03,   2.48869800e+02,\n",
       "          2.04986720e+03,   2.64561800e+02]),\n",
       " 'mean_score_time': array([ 0.01239996,  0.0086    ,  0.01200008,  0.00879989,  0.01160002,\n",
       "         0.0092    ,  0.01199999,  0.00839992,  0.01060004,  0.00920005,\n",
       "         0.01180005,  0.00880008,  0.01219993,  0.00880008]),\n",
       " 'mean_test_score': array([-0.77013412, -0.74650563, -0.7154283 , -0.71338944, -0.67804876,\n",
       "        -0.68963619, -0.67877562, -0.68184365, -0.68149827, -0.68206746,\n",
       "        -0.6837238 , -0.68351204, -0.68540385, -0.68488814]),\n",
       " 'mean_train_score': array([-0.76979967, -0.73533572, -0.71151394, -0.7016141 , -0.66802871,\n",
       "        -0.6740421 , -0.66378251, -0.66289682, -0.66290515, -0.66046145,\n",
       "        -0.66272745, -0.65958177, -0.66272783, -0.65956098]),\n",
       " 'param_C': masked_array(data = [0.001 0.001 0.01 0.01 0.1 0.1 1 1 10 10 100 100 1000 1000],\n",
       "              mask = [False False False False False False False False False False False False\n",
       "  False False],\n",
       "        fill_value = ?),\n",
       " 'param_penalty': masked_array(data = ['l1' 'l2' 'l1' 'l2' 'l1' 'l2' 'l1' 'l2' 'l1' 'l2' 'l1' 'l2' 'l1' 'l2'],\n",
       "              mask = [False False False False False False False False False False False False\n",
       "  False False],\n",
       "        fill_value = ?),\n",
       " 'params': [{'C': 0.001, 'penalty': 'l1'},\n",
       "  {'C': 0.001, 'penalty': 'l2'},\n",
       "  {'C': 0.01, 'penalty': 'l1'},\n",
       "  {'C': 0.01, 'penalty': 'l2'},\n",
       "  {'C': 0.1, 'penalty': 'l1'},\n",
       "  {'C': 0.1, 'penalty': 'l2'},\n",
       "  {'C': 1, 'penalty': 'l1'},\n",
       "  {'C': 1, 'penalty': 'l2'},\n",
       "  {'C': 10, 'penalty': 'l1'},\n",
       "  {'C': 10, 'penalty': 'l2'},\n",
       "  {'C': 100, 'penalty': 'l1'},\n",
       "  {'C': 100, 'penalty': 'l2'},\n",
       "  {'C': 1000, 'penalty': 'l1'},\n",
       "  {'C': 1000, 'penalty': 'l2'}],\n",
       " 'rank_test_score': array([14, 13, 12, 11,  1, 10,  2,  4,  3,  5,  7,  6,  9,  8]),\n",
       " 'split0_test_score': array([-0.77141467, -0.75593076, -0.72075786, -0.71795943, -0.68551756,\n",
       "        -0.69502472, -0.686109  , -0.68933112, -0.68850701, -0.68978247,\n",
       "        -0.6908726 , -0.69082677, -0.69236541, -0.69224586]),\n",
       " 'split0_train_score': array([-0.76932267, -0.73409045, -0.71169511, -0.70056136, -0.66730716,\n",
       "        -0.67250569, -0.66294063, -0.66145885, -0.66188199, -0.65934689,\n",
       "        -0.66188782, -0.65791242, -0.66195487, -0.65799778]),\n",
       " 'split1_test_score': array([-0.77141676, -0.7514851 , -0.71606152, -0.71824615, -0.67656527,\n",
       "        -0.6943463 , -0.67954986, -0.68560022, -0.68323259, -0.6858981 ,\n",
       "        -0.68657966, -0.68829038, -0.68903363, -0.68966472]),\n",
       " 'split1_train_score': array([-0.7687497 , -0.73444319, -0.70681707, -0.70072633, -0.66498018,\n",
       "        -0.67404042, -0.6613896 , -0.66270513, -0.66062368, -0.66055992,\n",
       "        -0.6605076 , -0.66020534, -0.66054013, -0.65976434]),\n",
       " 'split2_test_score': array([-0.76977863, -0.7454462 , -0.71559907, -0.71526195, -0.67951286,\n",
       "        -0.69041416, -0.67830044, -0.68066906, -0.68008119, -0.67890316,\n",
       "        -0.6805315 , -0.67908135, -0.6810758 , -0.68007719]),\n",
       " 'split2_train_score': array([-0.76998523, -0.73508748, -0.71125802, -0.70075848, -0.667539  ,\n",
       "        -0.6726128 , -0.66307141, -0.66183771, -0.66203075, -0.65885391,\n",
       "        -0.66186565, -0.65799814, -0.66177355, -0.65828552]),\n",
       " 'split3_test_score': array([-0.7682616 , -0.73502736, -0.71031046, -0.70406939, -0.6688026 ,\n",
       "        -0.67950998, -0.66974483, -0.67145482, -0.67303538, -0.67240353,\n",
       "        -0.67587159, -0.67512993, -0.67804615, -0.67745278]),\n",
       " 'split3_train_score': array([-0.77075217, -0.7372975 , -0.7147771 , -0.70398773, -0.67162983,\n",
       "        -0.67688804, -0.6674194 , -0.6664966 , -0.66636578, -0.6639262 ,\n",
       "        -0.66613089, -0.66300811, -0.66617527, -0.66301973]),\n",
       " 'split4_test_score': array([-0.76979881, -0.74463819, -0.71441229, -0.71140967, -0.67984608,\n",
       "        -0.68888556, -0.68017438, -0.68216311, -0.6826355 , -0.68335041,\n",
       "        -0.68476396, -0.68423198, -0.68649858, -0.68500016]),\n",
       " 'split4_train_score': array([-0.77018859, -0.73575999, -0.7130224 , -0.70203661, -0.6686874 ,\n",
       "        -0.67416357, -0.66409153, -0.66198582, -0.66362353, -0.65962032,\n",
       "        -0.66324529, -0.65878485, -0.66319533, -0.65873753]),\n",
       " 'std_fit_time': array([  3.31480657e-02,   3.23034747e-01,   4.23363732e+00,\n",
       "          6.57708592e-01,   4.20408312e+00,   3.38091806e+00,\n",
       "          7.21438676e+00,   1.23645871e+01,   1.85117516e+02,\n",
       "          3.27933198e+01,   1.92402245e+02,   3.76913150e+01,\n",
       "          2.13663521e+02,   3.67106670e+01]),\n",
       " 'std_score_time': array([ 0.00149664,  0.00048988,  0.00063241,  0.00039999,  0.00080001,\n",
       "         0.00074829,  0.00089442,  0.00048982,  0.00135643,  0.00039997,\n",
       "         0.00040007,  0.00039997,  0.00159996,  0.00039997]),\n",
       " 'std_test_score': array([ 0.00118579,  0.00707111,  0.00334849,  0.00526818,  0.00545632,\n",
       "         0.00556668,  0.00525591,  0.00599324,  0.00504102,  0.00598758,\n",
       "         0.00514196,  0.00577507,  0.00520985,  0.00557501]),\n",
       " 'std_train_score': array([ 0.00069612,  0.00113468,  0.00264887,  0.00129931,  0.00216592,\n",
       "         0.00158225,  0.0020131 ,  0.0018447 ,  0.00197528,  0.00181924,\n",
       "         0.00190929,  0.00190044,  0.00191821,  0.00183045])}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view the complete results (list of named tuples)\n",
    "grid.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.678048764431\n",
      "{'penalty': 'l1', 'C': 0.1}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果最佳值在候选参数的边缘，最好再尝试更大的候选参数或更小的候选参数，直到找到拐点。\n",
    "l2, c=100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEKCAYAAAA1qaOTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3XmczfX+wPHX55yZObMasxnDYOyy\nixuiUpGtjaK6qbiVvUUpRbcrqQgtKkWErhb9KnFbRURKtoQiZGeYMft61s/vj3NmDHPMnBnnzML7\n+Xicx/kun+/3fL6Nzvt8dqW1RgghhLhQhsrOgBBCiIuDBBQhhBBeIQFFCCGEV0hAEUII4RUSUIQQ\nQniFBBQhhBBeIQFFCCGEV0hAEUII4RUSUIQQQniFX2VnoCJFR0frhISEys6GEEJUK1u3bj2ttY4p\nLd0lFVASEhLYsmVLZWdDCCGqFaXUYU/SSZWXEEIIr5CAIoQQwiskoAghhPCKS6oNxR2r1cqxY8fI\nz8+v7KxUWYGBgcTHx+Pv71/ZWRFCVGGXfEA5duwYYWFhJCQkoJSq7OxUOVprUlJSOHbsGA0bNqzs\n7AghqrBLvsorPz+fqKgoCSbnoZQiKipKSnBCiFJd8gEFkGBSCvnvI4TwhASUcrhj7i/cMfeXys6G\nEEJUKRJQqoDQ0NDC7T59+lCzZk1uvPFGt2nHjBlD+/btadmyJUFBQbRv35727dvz6aeflukzt23b\nxrfffntB+RbiYnb4nns5fM+9lZ2NC1aRz1EpjfJKqUhgKZAAHAIGa63T3KSrD8wH6gEa6Ke1PqSc\ndTBTgUGAHXhbaz27YnLvW0888QS5ubnMnTvX7fm33noLgEOHDnHjjTeyffv2cn3Otm3b2LVrF336\n9Cl3XoUQoqjK6uX1FLBaaz1NKfWUa3+Cm3TvAy9orb9XSoUCDtfxoTiDTAuttUMpVasiMl0Rrr/+\netauXVuua/ft28fYsWM5ffo0ISEhzJ8/n2bNmvHxxx8zdepUjEYjkZGRfP3110yZMoW8vDzWrl3L\nM888w+233+7dBxGXpIJfwg3++34l58Rz2mZDm804LBa0xYI2m537OTloIG/nrqKp3dzAzbHzHXdz\nTLtN5/aG5fpse1YWAI68PAxBQe7Te0llBZRbgB6u7cXAWs4JKEqploCf1vp7AK11dpHTo4B/aq0d\nrnNJ3sjUc//7gz9PZJaa7s9EZxpP2lFa1qnBf25qdcF588Tw4cOZP38+jRs3ZsOGDYwdO5aVK1fy\n3HPPsXbtWmJjY0lPTycoKIhnn32WXbt28dprr1VI3oQ4l9YabDYcZgvaYi78MneYzWizBW0tsm+x\nOI9Zztk3m9FWy5lrXOcclvPsFw0YrgCC3V5iPg8NGlRB/0V8y5qYiKlRI59+RmUFlFitdSKA1jrx\nPCWMZkC6UupzoCGwCnhKa20HGgN3KKUGAMnAw1rrfe4+SCk1HBgOUL9+fe8/SRWRnp7Oxo0bue22\n2wqP2Ww2ALp168a9997LoEGDGDhwYGVlUVwEtNY4cnKxp6ZgT03Flprqek/DnpKC+cABsNs58sCD\nRb7Mz3yJn7VvsYDDUfqHlkIFBKBMJucrwB9DQMF2AMoUgCE4GGNExJl9kwnlX3CNaz8gABXg3FcB\nzmPJb81BKUXMY+PO+cDivR7P2xPS3XG3aT1M5zZZyZ99atp0APxr13afzot8FlCUUqsAd08wycNb\n+AFXAR2AIzjbXIYCCwATkK+17qSUGgi850pbjNZ6HjAPoFOnTucpHzp5WpIoKJksHdHVo/QVQWtN\ndHS02zaVd999l19//ZUvv/ySdu3asWPHjkrIoaiKCgNEWir2lBRnYEhLxZZSEChSsKemnRU8tMXi\n9l4qOBhsNjAasWdlYggwYQgNxVjwxR1gKvLlf+aL+9wvchVgOvvLv/ALv8iXf8Fxf3+fdWtP+/Aj\nAMKuvdYn968oxho1ADAEB/v8s3wWULTWPc93Til1SikV5yqdxAHuqqyOAb9prQ+4rvkC6IIzoBwD\nPnOlWwYs9Grmq6GIiAji4uJYtmwZAwYMwOFwsHPnTtq1a8eBAwfo0qULnTt3ZsWKFRw/fpywsDCy\nXHWronJ5s91Ba43OzS1ScijynpLqDBau0oQtzRkotNns9l4qOBi/iAiMkZH4xcRgat4cv6hIjBGR\nzmOubb9IZxpDUBCHe3UANA2WLr3gZ6l0J3dWdg68owKfo7KqvFYA9wHTXO/L3aTZDEQopWK01snA\ndUDBYiZfuPbfA64B9vo8xxXkqquuYs+ePWRnZxMfH8+CBQvo3bu3R9d+/PHHjBo1ismTJ2OxWBgy\nZAjt2rVj3LhxHDx4EK01N9xwA61btyY2NpYZM2bQoUMHJk2aJI3ylamE/+ELA4Try9+W4iwxuCs5\n2NKcQeO8ASIoyBkgoqIwxkRjat4cY2QEfpGRGCOjCgODcz/S5w24VV2Df9ap7Cx4RUU+R2UFlGnA\nJ0qp+3FWZw0CUEp1AkZqrR/QWtuVUuOB1a5uwluBd4tc/4FSahyQDTxQ4U/gRdnZZ/obrF+/3qNr\nEhIS2LVr11nHGjVqxHfffVcs7YoVK4odi4mJkcXGqgCtNXazA4fZwamXZ7hKDs7AYEtLxZ6ahj7P\ntDcqMLDwy98YHYWpadOzSw5RrnOuUkSZqzzsNrDlg80MdvOZ7cJX0XPO/QY3KWcPo43vgHYA2vmu\nC96LHsPNMX3OfsExXcq9il5X2r2KpivhXqdc/3+9f0t5/rRVR8FzpB+FmvV8+lGVElC01inA9W6O\nb6FIcHD18GrrJl060N+XeSxJVWo7EdWTLSWFjBX/I+PzzzAnOdsk0j780FViiMIYWRNTwwYYI2rg\nVyMUY3goxhpB+IUFYgwNxC80AIMfzi91u+WcL/s0sJ0EixmOmeFQPtiKpil6zTnnigQH55dsOX3r\nbhRAaRQog7MxWRmcr3OPoYqcV+7TnJXu3DTnu87NMe3q/WXNK/9/h6qg4Dnc90X2qkt+tmEhKoq2\n2cj+/mvSP/2E7I2/gd1BUL1QottkEd4gD//IEJTjhPPLvIAVSHG9ysovEIwm8DM5t/0CXO8m5/GA\nUAiOdp0vks5YkO6ca846V/S+pjOf8+kDzi/jf35chi9zw3l6PlWyha7frMO+qtx8XKiC56jp+16u\nElCE8Lb8TEj9G1L+htQDmHfvIGPDX6TvysKepzCa7EQ2zaNmozxM9f3JO2jDnB1EwPV3nf8L2+2X\necHxc9OanGkr40vaL8D5HhxZ8Z8tKp0EFCHKw5ILqQfOBI6Uv89s5yRhtyoyjwSRcTCYvNMBoCC0\nZSw1r+tIaI/rULWaQUQD8DOR1KsDAGF9p1fyQ4mzVPeSSYEKfA4JKEKcj80MaYeKBIv9haUOMo+f\nnTY0Fh3RiDy/f5B+OJvMbYfQZisBjRpSa9jthN98E34xMW4/5mLpTQRcPF/ColwkoJTHxVK3Wp15\n629gt0H6YWeQKAwYrpJGxtGzG6aDIiGqMSRcBVFNIKoRRDbGag0h45sfSP/kc6yHf8MQEkL4LQOo\nedtAAtu2LX3gnfw7EhcJCShVQGhoaGHX4T59+rBx40a6d+/Ol19+WSztmDFj2LBhAxaLhYMHD9K8\neXOAMk3wuGzZMvbv388TTzzhvYeoyhwOyDzmqpra7woersCRdggctjNpTTUgshHEd4J2d0JkY2cQ\niWx0VruAw2Ih+4cfSH99DjkbNoDDQfAVVxAzejRhN9xwyY/hEJcmCShVjLemr7fZbPj5uf/zDhgw\nwDuZrUq0hqyTxaumCt7tRQb7+Qc7A0StlnDZza6A0dhZ6giJLrExO3/3btI/X0bmihXYMzLwq12b\n6JEjCB8wgIB6vu3jL0RVJwGlirmQ6eu7d+/ONddcw/r16xk4cCANGzbkxRdfxGKxEBMTw5IlS6hV\nqxbz588vnGl4yJAhREVFsXnzZk6ePMmsWbOqR8Bx2CHrBHxynyuIHABrzpnzxgCIaOgMEk17nilp\nRDWBsLgy9YCyp6eT8eVXpH/+GeY/d6P8/Qnr1ZPwgbcR0rULymj0wQMKUf1IQCnqm6c8m/fmpGty\nxYUejK2s3Qb6TruwfJVBZmYm69atAyAtLY2bb74ZpRTvvPMOs2bNYvr04j2JkpKS2LBhAzt37mTw\n4MHVI6CkHYLsRDAYnUGiQfczVVNRTSA83nmunLTdTs4vG8n4/DOyvl+FtloxtbyM2GeeIfzG/hhr\n1vTeswhxkZCAcpG58847C7ePHDnC4MGDOXnyJGazmWbNmrm95tZbb0UpRdu2bTl+/LjbNFXKgR+d\nwSSsDjz8m1dvbTl6lPTPPyfji+XYEhMxhodT8847qTlwAIGXXebVzxLiYiMBpShPSxJVuJdXSEhI\n4faYMWOYOHEi/fr1Y9WqVUyb5v75TCZT4bbb1eOqEnMWLB8LfkFQs4FXbunIyyNr5UrSP/uc3E2b\nwGAgpFs3Yic8Seh112EICPDK5whxsZOAchHLyMigbt26aK1ZvHhxZWfHO1b+29mdN7bNhVVpaU3+\n7787G9i/+gpHTg7+9esT8+ijhN96S4UsRiTExUYCShVzIdPXn2vy5MkMGDCA+Ph4rrjiChITE72c\n2wr29w+wdSFc+RDcMLVct7CdPk3G8hWkf/45lr//RgUFUaN3b2reNpCgTp18tliTEJcCVeWrOLyo\nU6dO+twp23fv3s1lZa0br8JVXr5Srv9O3pSfCXO6QkAwjFgH/p6P89BWK9nr1pH++TKy164Fu52g\n9u0Jv20gNfr2xRga6rt8C3ERUEpt1Vp3Ki2dlFDK4xIKJFXGyknObsL3f+9xMDH//Tfpn31OxvLl\n2FNSMEZHEzVsKOEDB2Jq1MjHGRbi0iMBRVR9+1fBtveh26POEewlsGdnk/n112R89jl5v/8Ofn6E\n9riGmgNvI/Sq7ih//wrKtBCXHgkoomrLS4flD0FMC+jxdOHhb3reCkDfVV+gHQ5yN28h4/PPyfzu\nO3R+PgFNGlNrwgTnpIxRUZWVeyEuKRJQRNX23STIPgV3LgH/wLNOGW1WTr/9NumfL8N69CiG0FDC\nb7nFOSljmzbSwC5EBZOAIqquvSth+xK46nGo2/GsU6HZ6USlnyL59dkEd+lCzENjCevVq1pOynjH\n3F8AWVpaVH8SUMph2LfDAFjYZ2El5+QilpcG/3vYOYHjNWevT56ycBHR6afINYXQ5qsvCIiPr6RM\nCiGKMlR2BoRz+nqA7du307VrV1q1akXbtm1ZunRpsbRjxoyhffv2tGzZkqCgINq3b0/79u359NNP\ny/SZ27Zt49tvv/VK/n3i26chOwluneNc0hbnYMTkN94kafp0coJCSYquK8FEiCpESihVSHBwMO+/\n/z5NmzblxIkTdOzYkd69e1OzyESEnk5fX5pt27axa9cu+vTp45W8e9Vf38DvH8HVT0Id5/K4WmuS\npk0ndfFiwgcMYEr9fjgMRvpWclaFEGdICaUKadasGU2bNgWgTp061KpVi+TkZI+v37dvH71796Zj\nx45cffXV7N27F4CPP/6Y1q1b065dO6699lry8vKYMmUKH3zwQblKNz6Vmwr/ewRiW8PVzgXAtN3O\nyWefJXXxYiKGDCHuhak4LmDalarmUMBMDgXMrOxseMWwb4cVVgmLS4+UUIqYvmk6e1L3lJquII0n\n/+O0iGzBhCsmlJruXJs2bcJisdC4cWOPrxk+fDjz58+ncePGbNiwgbFjx7Jy5Uqee+451q5dS2xs\nLOnp6QQFBfHss88WrolSpXwzAXJT4O5PwS8AbbVyYsIEMr/+hqiRI4h55BHpvVWF/ZmYWdlZEJVI\nAkoVlJiYyD333MPixYsxGDwrRKanp7Nx40Zuu+22wmM2m3Np227dunHvvfcyaNAgBg4c6JM8e8Xu\nL2HnJ87xJnFtceTnc/zRcWSvXUut8Y8T9cADZyW3OzRZ+Vb8jQYCjAYMBgk0QlSmSgkoSqlIYCmQ\nABwCBmut09ykqw/MB+oBGuintT6klLoemIGzyi4bGKq13n+h+fK0JOHLXl6ZmZn079+fqVOn0qVL\nF4+v01oTHR3ttk3l3Xff5ddff+XLL7+kXbt27Nixw5tZ9o6cFPjyUeeCZFc9jiMnh6Ojx5C7aRO1\n//MsEXfddVby1BwL+5OyaTN5ZeExP4NyBhc/gyvIqMLtguMBheeVm2NF051zL1c6fz9FgNHovN7P\ngMlowL/gnNFAQMF5P+VK77q/BLxqR3pzll1llVCeAlZrracppZ5y7bv7Nn8feEFr/b1SKhRwuI6/\nDdyitd6tlBoNPAMMrYB8+5TFYmHAgAGFpYmyiIiIIC4ujmXLljFgwAAcDgc7d+6kXbt2HDhwgC5d\nutC5c2dWrFjB8ePHCQsLIysry0dPUg7fPOEcFX/PF9izczk6fAR5u3ZRZ/o0wm+++ayk+5Oy+Ts5\nm+AAI4/2bIbF7sBic2C1O18WmwOLXRdunznm3M612LAWOX/29brwmLcZDQp/oyoMYAUB53TWAJTB\nxv2LNhMTZjrzCjUR7XqPCTMRYpIKBVG1Vda/0FuAHq7txcBazgkoSqmWgJ/W+nsArXV2kdMaqOHa\nDgdO+DCvFeaTTz5h3bp1pKSksGjRIgAWLVpE+/btPbr+448/ZtSoUUyePBmLxcKQIUNo164d48aN\n4+DBg2itueGGG2jdujWxsbHMmDGDDh06MGnSJG6//XYfPlkp/lwOuz6Da5/B5lebI/feh+XAAeq+\n9io1evU6K2m22cbIJVsxKEXT2DAevNo3kzxqrQuDztkBSRcGH/M5QcxqdwayokHMecx9wCo4dnL/\nH2iHP4kZ+ew4nkFKthmHm0nAgwOMhYGmaNCJCTMRXeRYdKiJAD/pb3OhpD2o7CoroMRqrRMBtNaJ\nSqlabtI0A9KVUp8DDYFVwFNaazvwAPC1UioPyAQ8rxuqgrKznbFyyJAhDBkyxKNrEhIS2LVr11nH\nGjVqxHfffVcs7YoVK4odi4mJ4dyp/CtFzmn48jGIa4+10R0cGXIP1pMniX/7bUK7dzsrqdaaCZ/u\n4EByNs1iwzD58EtTKeWsvqqAL+bOC51ru3w9bATgbBtKzbGQnGUmOdtMcpaZ0673gte+pGx+/juF\njDyr23vWDPYvNfDEhJmIDA6QqriLXEVW3fksoCilVgHulr2b5OEt/ICrgA7AEZxtLkOBBcA4nO0p\nvyqlngBewRlk3OVjODAcoH79+mV4gvOTOlUv+upxMGdi6fRvDt83FEdmFvUXzCe4Y8diSRf8dJCv\ndibyVN8WrNmTVAmZrRhGgyr8wi+N2WbndLalMNCcG3iSs838diSdpKx88q3Fq/GMBkVUSECxwOMu\n+ISZ/KSHXTVUkSUtnwUUrXXP851TSp1SSsW5SidxgLtvh2PAb1rrA65rvgC6KKVWAO201r+60i0F\nzjvkW2s9D5gHzgW2yvc0wid2fQ5/fkF+87EceeQ5sNupv3gRQa1aFUv664EUXvpmD71bxTLi6kYX\ndUApC5Ofkbo1g6hbs+Q5zLTW5FjsJQae5CwzexKzOJ1txuamzs3kZyg18NitoRiMeb56XFHFVVaV\n1wrgPmCa6325mzSbgQilVIzWOhm4DtgCpAHhSqlmWuu9QC9gd8VkW3hNdhJ89Th5hjYcfW01ymSi\n/qKFmJo0KZb0VGY+Yz78jQaRwcwY1O6i+pWcZcnCRjYaC/N3zsegDBgwoJRybisDiiLbSmGgyPa5\n51Fnrj3PfZRJERloIDraQMvC834o/DGoGqAVORYH6blWMvNspOXYSM+1kZZrJS3HTFpONn+nWdl8\nzEp6rg1QZ166L1r70fyZL4kMCaJmcACRIf5EBAc4XyEBRAT7ExkS4DwXHEBN135wgPGi+tteiior\noEwDPlFK3Y+zOmsQgFKqEzBSa/2A1tqulBoPrFbOf2VbgXe11jal1IPAZ0opB84A86/KeQxRLlrD\nV4+Rc9TMsQ05GKOiqb/wPQLq1SuW1Gp3MOaDbeSYbXz4YGdqBFbvBbKsdiu/J//OxsSN/JL4C7tO\n78JhcFZFvb7t9UrOnYf8gZrOV0mLJ+fiTz6BnHKYcOQFYM8KwGYLQDtM4HC+O7ed70ZMhPiHEBYQ\nQrgplIigMCKCQokOqUFMSA1iQoMLg5AzOPkTKtVwVUqlBBStdQpwvZvjWyjSFuLq4dXWTbplwDJf\n5lH40K7PyF69kmO/1MK/fl3qv7cA/9hYt0lf/Ho3Ww6nMfuuDjSLDavgjF44rTX70/fzy4lf2Ji4\nkS2ntpBny8OgDLSObs2DbR5k0favMBDI+ns/xqEdaK1xaAcOimxrB5oi21rj4Mx2wTm315bhPp6c\nL+l+036eC2gevHwQedY8cqw55NhyyLXmkmvNJduaQ5YlixzXfr49F82Zth0LkOJ6Ycc5yszVv1M7\njK7gE1AYiNAm/FUQJmMggcZgQvxDCPEPoYYphPDAUCICw4gKDiU6OJxaoTWIDQsnNrQGIf4hmIwm\nCUZeJh3by+HwPfcC0OC/71dyTqqhrFNkvvUkx3+KwtS8GfUXzMcvMtJt0uXbj7NwwyGGdUvg5nZ1\nKjij5Xcq5xQbEzcWvk7nnQYgoUYCtzS+ha51uvKP2v8gLMAZIP+7fQ0AJmPpjfBV3ayfnTNkj2o3\nyqP0WmvMdjM5VlfQseU6g5A1h1ybKwhZsknLzyY1L4u0vCyyzDlkWZxp8uy55NtTsTjyyNT5pDny\nwWJ3RqbShllpAwYC8VMmAlQQJmMwQX5BhPiHEBoQSp49EwXM+PUNagaGUjMwlGD/YIL9ggn2dwav\ngu0gvyCC/YPxN1TvEvSFkoBSBYSGhpKdnc327dsZNWoUmZmZGI1GJk2axB133HFW2jFjxrBhwwYs\nFgsHDx6kefPmADzzzDMejyVZtmwZ+/fv54knnvD6s5RIa9Kfv5fEHwMIan0Z9d5bjDHMfanjr5NZ\nPPXZTjo1iGBiv8uKna9Ki1FlW7LZcmpLYSnkQMYBACIDI+kS16XwFRcaV8k5rXqUUgT6BRLoF0hU\nkHeWarbYLeRYcjiVk8mprHROZWdwOieblNxM0vKzyMjPJtOSQ7bFGcTy7Lnk2nPJ0PmgMsGQjDJY\nwGBGGcy8v2eex59twI8AQxAmYxBBfkEE+QUT4h9MqH8wNUyhhJlCCPUPOSswFQSjgv3CgOUXUhik\nqktJSgJKFeLN6ettNht+fu7/vAMGDPB+5j2QOv1xTq04QkiresQvXoIhONhtusx8K6OWbCU00I85\nd1+Ov7H4WJDKnBbD6rCyM3mnsx3kxC/sPL0Tu7YTaAykY+2ODGw6kC5xXWga0RSDKn0cS8u4GqWm\nEZ4LMAYQEBRARFAELaIbeHyd1ppss430XCupORbuW/4s2h7AE53HuEpH2YXBKMucQ7Y1hxxrLnm2\nXPLteZjteTgwk2+woAxmMFhcgek0qnDbGaSUwf34IXcMykigMahIySiY0ABX6ahIUAo5J1AVnLeT\nh8KA2W72eSlYAkoV0qxZs8LtotPXFw0oJenevTvXXHMN69evZ+DAgTRs2JAXX3wRi8VCTEwMS5Ys\noVatWsyfP79wpuEhQ4YQFRXF5s2bOXnyJLNmzfJ6wNFak/L6TJIXfUNokyDqfrACQ6D7bq5aa8Z/\n8juHU3P56MEu1KoR6DZdRdJacyDjQGEJZPPJzeTacjEoA62iWvGv1v+ia52utItpR4AxoLKzK8pJ\nKUVYoD9hgf7UiwzGFHIcgCFdGnp0vdYas81BVr6NrHyr691GttlKpms7K99Kdr6NzDwL6eZsMs05\nZJqdpSVncMrD4shzBSNzYRDKN1jIKBKkDIbTGP0shYFKKzMOZcY5icg5XL9p9qUcpnWtZsXPe5EE\nlCJOvvgi5t2lT1+fv8eZpqAtpSSmy1pQe+LEMuelPNPXg3NyyXXr1gGQlpbGzTffjFKKd955h1mz\nZjF9+vRi1yQlJbFhwwZ27tzJ4MGDvRpQtNYkzZhJ6nvvUaOhmTrzv0CdJ5gAvPPjAVb+eYpn+l/G\nFQ3dt61UhOTc5DPtICc2kpTnHPfSoEYDbmp8E13jutKpdifCTeGVlkdRtSilCPQ3Euhv9GhQ6vnY\n7A6yzc4AlOkKQFn5NrLMrmBUNDiZz2xn5lvJMueSZXa2P2lVEJCcwUnbPPtheiEkoFRB5Zm+vsCd\nd95ZuH3kyBEGDx7MyZMnMZvNZ5WAirr11ltRStG2bVuOHz9+QXkvSjscnJwyhfSPl1KzSQ61n56A\nqt38vOk37D/NjO/20L9tHPd39+xXobfkWHPYemprYSlkf7pz8uoIUwSd4zrTtU5XusR1oU5o9ekc\nIKonP6OBmsHOcTrlVTCQNTvfRv+PxuBw+NM0xjttVCWRgFKEpyUJX/byKu/09QVCQkIKt8eMGcPE\niRPp168fq1atYtq0aW6vMZnO/JrS2juTCWibjRMTJ5K54n9EtbYQ07clquv5e/6cSM/j4Y9+o1FM\nKC/f1tbnjZA2h41dp3fxS+IvbDyxkR3JO7BpGyajiY6xHbm58c10rdOVZhHNPGoHEU7SHlQ1KKUI\nNfkRavLDz5QOQKC/71c5lYBShVzI9PXuZGRkULduXbTWLF682As59IzDYuH4Y4+RvWo1MT1qERW/\nF3XrW3Ce0pbZZmf0B9vIt9p5Z0hHn0zTrrXmYOZBNp5wDijccnIL2dZsFIqWUS0Z2nooXeK60L5W\n+4ui+64QlUECShVyodPXn2vy5MkMGDCA+Ph4rrjiChITE72YW/ccubkcG/sQOT//TOx9vYg0L4Ze\nMyDq/G1Bz3/5J9uPpvP23ZfTpFZJY6/L5nTe6cI2kI2JGzmVewqA+NB4+jbsS5e4LnSO6yztIEJ4\niQSUKsBb09f/9NNPZ+3fdtttZy0JXOCBIkvpLlmyxG1eysOemcnRESPJ+/134v49nppHJkPCVfAP\ntxNBA/DZ1mMs2XiEEVc3om+bCxunkWvNdbaDJDrbQfal7QMg3BRO59qd6VLHOR6kXljxKV4q08U0\ne/XF9Cyi7CSglIOMkC/OlprKkQcewLxvP3VfmUWN5LngsMPNb5y3quuPExlMXLaTLo0ieaL3+Rvr\nz0drzY7kHYUN6duTt2Nz2Aj+T8ZDAAAgAElEQVQwBNAhtgOPXv4oXep04bLIy6QdRJSZtAeVnQQU\nccGsp05xZNi/sB4/Tr233iQ05ABs/AH6z4JI9721MnKtjFqyjZrB/rxx1+X4uRm8WJLU/FQOZx7m\n7q/vRqFoEdmCe1reQ9e4rnSo1YFAv8ofvyKqNyltlZ0EFJy/dKvL1AaVoaSeX5ajRzky7F/Y09Ko\n9+48QprVhjn/hIZXQ0f3k0A7HJpxn2wnMSOPj4d3LVOffYd28OZvb3Ig4wAh/iFMvnIynWt3JiIw\noszPJYTwrks+oAQGBpKSkkJUVJQEFTe01qSkpBAYWPwXv3n/fo4M+xfaYqH+ooUEtW4N798CaLj5\nzfNWdb21Zj8/7Eliyi2t6NjA80CQbcnm6fVPs/bYWqIDo6lfoz59EvqU99GEuCQkWMZX2Gdd8gEl\nPj6eY8eOkZycXNlZqbICAwOJj48/61jeH39w9P4HwN+P+v99n8BmzWDzAjj4I9z4GkS4n0Ppx73J\nvLJqL7e2r8M9XTyfZ+lw5mEe/uFhDmce5ukrnmbloZXyA0AID1TkRKqXfEDx9/enYcOKHZVd3eVu\n3crRESMx1qjhXBirQQNIOwQr/w2NroWOQ91edzQ1l0c+/o3msWG8OLCNxwHh5+M/M37deIzKyLxe\n87gi7gq+P/y99x5ICOEV0vVFlEn2Txs4cv8D+EVH0+CDJc5g4nDA8rGgDM5eXW4CRb7VOXjR7tC8\nM6QjwQGl/5bRWrP4j8WMWj2K2iG1+aj/R1wRd4UvHksI4QWXfAlFeC7z++858djjBDRuTP357+IX\nHe08sWUBHFrvDCY13Y/xmLziD3Yez+DdezuREB3iNk1RZruZKb9MYcXfK+hZvycvdH+BYH/3090L\nIaoGCSjCIxnLl3Ni4iSCWrem3ry5GMNdo8tTD8L3z0KTntDhHrfXfrzpCB9vPsqYaxvTq6X7pX6L\nSspN4tE1j7Lz9E5Gtx/NiLYjZByJENWABBRRqtQPP+TUlOcJ7tKFem+9iaFgAkqHA5aPAYM/3DTb\nbVXXzmMZPLviD7o3ieaxXqUPXvw9+XfGrRlHtjWb13q8xvUNrvf24wghfEQCiijR6XffJXnWK4Re\ney11X3sVQ5GZidk0Dw5vgFvmQHjdYtem5VgYuWQrMaEmZt/VAaOh5Eb45fuX89wvz1EruBZLei2h\nWYRvFwMSQniXBBThltaa5NdeJ2XuXGr060ed6dNQ/v5nEqT8DasmQ9Pe0P6fxa63OzSPLN1OcpaZ\n/xvZlciQ86/tYHPYmLVlFkt2L6Fz7c7MvGYmNQN9vxiQEMK7JKCIYrTDwakXXiTtgw+oOeh2ak+e\njDIWWUvBYYcvRoNfANz0mtuqrtdX7WXd3mReHNCGdvXOHxwyzBk88eMT/JL4C3dfdjfjO43Hz1D6\nP0uZFkOIqkcCijiLttlIfObfZHzxBZFDh1JrwpPFx4v8+g4c3QgD5kKN4isYrt59itk/7GdQx3ju\nuuL8M/vuT9vPw2seJjEnkSlXTmFAU++uZS+EqFgSUEQhbbFwfPwTZK1cSfRDY4kePbp4MDm9H1ZP\ngWZ9oe0dxe5xOCWHcUu306pODZ6/tfV5By/+cOQHnl7/NEF+QSzsvZD2tcq35osQouqQgCIAcOTl\ncezhR8hZv55aT00gauhQN4ns8MUo8At0W9WVZ7Ezcsk2lFK8M6Sj2yVHtdbM2zGPN7e/SauoVrx2\n7WvUDqnto6cSQlSkSuncr5SKVEp9r5Ta53ovNkOgUupapdT2Iq98pdStrnMNlVK/uq5fqpQ6f4uv\nKJU9O5sjDz5Izk8/UXvKc+6DCcAvb8GxTdBvJoSdHQS01kz6Yid7Tmby2p3tqRdZfBBirjWX8T+O\n583tb9K/UX8W9VkkwUSIi0hljRZ7ClittW4KrHbtn0VrvUZr3V5r3R64DsgFVrpOTwdedV2fBtxf\nMdm++NjS0jgydBh523+nzswZRAwe7D5h8l74YSq0uBHa3F7s9Ae/HuHzbcd55PqmXNu8VrHzx7OP\nc88397DqyCoe7/g4L3V/SdYsEeIiU1kB5RZgsWt7MXBrKelvB77RWucqZ6X8dcCnZbheuGFNSuLI\nvfdi3ruX+DdmE96/v/uEdht8MRICQuDGV4tVdf12JI3n/vcHPZrH8PB1TYtdvvnkZu768i4SsxN5\n6/q3GNp6qMwULMRFyKOAopTqppQKcW0PUUq9opTyfO7x4mK11okArvfiP2nPdifwkWs7CkjXWttc\n+8eA4qPqRIksx45zeMg9WI6foN68uYRde+35E//yBhzfCv1nQujZf6rT2WZGf7CN2BqBvHZHewzn\nDF5cumcpw1cOJ9wUzof9P6R73e6+eBwhRBXgaaP820A7pVQ74ElgAfA+cM35LlBKrQLcVZBPKksG\nlVJxQBvgu4JDbpKdd0lBpdRwYDhA/fr1y/LRF6XD99yLIy8P2+nTOHJzafDeAoLal9DDKmkPrHkR\nWt4CrQaedcpmd/DwR7+RmmPhs1FXUjP4TFOW1W7lpU0v8X97/4+r469m2lXTCAsI89VjCSGqAE8D\nik1rrZVStwCva60XKKXuK+kCrXXP851TSp1SSsVprRNdASOphFsNBpZpra2u/dNATaWUn6uUEg+c\nKCEf84B5AJ06dTr/WraXCEduLvl//YUxPJwG7y8msEWL8ycuqOoyhUG/WcWqumZ9v5ef/05hxu1t\naV03vPB4Sl4Kj619jG1J23igzQOMbT8Wo6F4jy8hxMXF0zaULKXU08AQ4CullBHwL+WakqwACgLS\nfcDyEtLexZnqLrRzgfM1ONtVPLleuJgPHCB/zx6UUjT4739LDiYAG16DE79B/1kQGnPWqW93neTt\ntX9z1xX1GdTpzODF3Sm7ufOrO/kz5U9evvplHrn8EQkmQlwiPA0odwBm4H6t9UmcbRYzLuBzpwG9\nlFL7gF6ufZRSnZRS8wsSKaUSgHrAj+dcPwF4TCm1H2ebyoILyMslwZaSwtHhI0ApTC1aYGpUyiqV\np/6AtdOc1Vytzh7BfiA5m/H/9zvt4sOZfHPLwuPfHPyGe7+5F4DFfRfTt2Ffrz+HEKLq8rTKKwtn\nVZddKdUMaEGRUkNZaa1TgGLzkmuttwAPFNk/hJsGd631AUCW7vOQIz+fo6NHYzt9GlPTphgCS+mu\na7c6BzAG1XSOOSki12Jj5JKt+BsVc4Z0xORnxO6w88Zvb7Bg1wIur3U5s3rMIjoo2odPJISoijwN\nKOuAq1wDEFcDW3CWWu72VcaEd2iHgxNPTiB/x07qzn6dGr16lX7RT69C4u9wxxIIiTpzL6156rOd\n7EvK5v1/XUHdmkFkWbJ4av1TrDu2jtub3c7EKybib7yQ2lAhRHXlaUBRrjEg9wNvaK1fVkpt92XG\nhHckzZxF1sqV1JowwbNgcnIn/PgytBkEl9101qlFPx9ixe8neKJ3c65qGsPhzMM89MNDHM08yqTO\nk7ij+R0yvkSIS5jHAUUp1RVniaRgVLq0tFZxaR99ROp77xHxz38SObTETnlONourqisC+r581qkt\nh1J54avd9LwsllHXNGbD8Q08se4J/JQf826Yxz9q/8NHTyGEqC48DSiPAk/j7L77h1KqEc6eVqKK\nyv7xR04+P5XQa64hduLTnpUc1s9yllDu/BCCIwsPJ2XlM/qDbcRHBDFzUFve/3Mxr257lSY1mzD7\nutnUDZVxpUIIDwOK1vpH4EelVJhSKtTVKP6wb7Mmyiv/zz85Nu4xTC2aU/eVWSg/D/7Mib/D+pnQ\n9k5ocWYKFqvdwdgPfyMz38q797Vj+tb/8OWBL+nVoBdTu00l2L/4JJBCiEuTRwFFKdUG58j4SOeu\nSgbu1Vr/4cvMibKznjzJ0ZGjMIaHU+/tdzCEhJR+kc0Cy0ZBcDT0nXbWqenf7GHTwVSeG1iXadsf\nZlfKLsa2H8vwtsOlvUQIcRZPq7zmAo9prdcAKKV6AO8CV/ooX6Ic7NnZHB0xEkdODg0+/BD/2NKm\nSHNZ9zIk/QF3LXW2n7h8ueME8386yI3/MLPo0Dhyrbm8fu3rXFf/Oh89gRCiOvM0oIQUBBMArfXa\ngskiRdWgrVaOPzoO8/791Js7l8DmzTy78MRvsP4VaH83NO9TeHh/UhZPfrqDxo3+5Ofcj6gdUpt3\ne71Lk4gmPnoCIUR152lAOaCU+jfwX9f+EOCgb7IkykprzcnnpzoXyHp+CqHdu3l2oc3srOoKjYXe\nLxYezsq38uB/NxFQawVJpvV0ie3CzGtmEm4KL+FmQohLnadTr/wLiAE+B5a5tof5KlOibFIXLCD9\nk0+IevBBIgYNKjHtsG+HMexb159u7TRI3g03z3aOiscZnMb938+cCnoTe9h6hlw2hLd7vi3BRAhR\nKk97eaUhvbqqpMxvvyVp5ixq9OtLzLhHS7/g5E7n+/GtzskfO9wDTc8MeJy+ei2/5E/GPyST57o9\nz61NZO0yIYRnSgwoSqn/UcJaI1rrm72eI+Gx3G2/ceLJCQRdfjlxL72EMnhW4PRzOJxVXWF1oPcL\nhcfnbPqCJUefJzAgmAV9F9K+VgnrpAghxDlKK6HMLOW8qCSWw4c5NmYMfnG1iX/rTQwmk8fXDkhP\nhcw0GPI5BIbj0A5mbXqL9/fMw9/egP+7dR6NIur4MPdCiItRiQHFNaBRVDG2tDTnVPRaU3/uXPwi\nIkq/yKVRfh59MtOg41Bocj251lwmrp/E6qOrcGRdzpKBs2gUITMFCyHKztOBjTspXvWVgXPW4amu\n6ehFBXBYLBx76CGsJ05Qf9FCAhISPL/YbuP+lFOkGv2IvmEqx7KO8ciaR9ibto/8U/15pfdDtKoj\nwUQIUT6edhv+BrADH7r278S5tnsGsAi4yf1lwpu0w0Hi0xPJ27KVOrNmEtyxY9lusGMpdaxWXq8V\nR9fU3Tz+4+Pk22zkHhnG0A69uamdzMklhCg/TwNKN6110cENO5VSG7TW3ZRSQ3yRMVFc8uzZZH71\nFTHjxhHev3/pFxRlt8G6Gez3D+B/AQEs/H44tYPrkbx3EB1rNeWpvqUsByyEEKXwdBxKqFKqc8GO\nUuoKINS1a/N6rkQx6Z99Rso7c6k56Haihj9Y9hvsWIoj7SBPxsRwyl/TJa4buYdGUcNYhzfv7oC/\n0dN/CkII4Z6nJZQHgPeUUqE4q7oygftd06+85KvMCaecn38m8T+TCenWjdrPPlv2SRldpZNX6zVj\nn18+kTaF7cR9nEg9zUfDL6dWWClLAgshhAc8Hdi4GWijlArHuXpjepHTn/gkZwKA/L17OfbwI5ga\nNqTua6+i/MuxvO6Oj/nAnswiv0j8dDgpaT05fDqZZ29syT8SIku/XgghPOBRPYdSKlwp9QrO9eRX\nKaVmuYKL8CFrUhJHR47EEBREvbnvYAwLK/tN7FZW//Iy0yMjubbetTiy25N9uiM3tavDsG4JXs+z\nEOLS5WnF+XtAFjDY9coEFvoqUwIcubkcGzUae3oG8e+8jX+d8g003P7LK0wIstMmrD7PXDGVzMQe\nGAMymDawjaxnIoTwKk/bUBprrW8rsv+cUmq7LzIkQNvtHH98PPm7dxP/1psEtWpVrvscStvPQ/uW\nEIsfb/R7n4XrE9H2IMLjVxJiks55Qgjv8rSEkqeU6l6wo5TqBuT5Jkvi1EvTyF6zhthJEwm79tpy\n3SMlL4VR3w5DaTtvX/4kNlsI89cfxBR2AP9AGYcqhPA+T0soo4DFBY3yQCow1FeZupSlvv8+aUuW\nEHnffUTefXe57pFrzWXs6jGcNqezwB5B/bb/5NkVf2C1OwiP3ublHAshhJOnvby2A+2UUjVc+5k+\nzdUlKmv1ak69NI2wXj2p9eQT5bqHzWFjwroJ/JnyJ68lJdP21pkcSc3jw1+PcMc/6rEqXf50Qgjf\nKG36+sfOcxwArfUr5flQpVQksBRIAA4Bg11rrhRNcy3wapFDLYA7tdZfKKU+ADoBVmATMEJrbS1P\nXqqKvJ07Of74eALbtKHOyy+jjMYy30NrzbRN01h7bC2TchXX1mwBTW9g1tLt+BkVD1/flFWf+SDz\nQghB6W0oYaW8yuspYLXWuinOrshPnZtAa71Ga91ea90euA7IBVa6Tn+AM8C0AYJwDrystizHjnN0\n1Gj8oqKoN+ctDEFB5brPgl0LWPrXUv4V05k7Tx2GHk/zR2Imy7efYFi3hsTWCCTBMp4Ey3gvP4EQ\nQpQ+ff1zPvrcW4Aeru3FwFpgQgnpbwe+0VrnuvL1dcEJpdQmIN4nuawA9sxMjo4cgbZYqLd4EX7R\n5Zvt98sDX/L6ttfpm9CHR7Z/B3U6QNMbmLFoM+FB/oy8pjEAS0d09Wb2hRCiUJkncFJKeaNVN1Zr\nnQjgeq9VSvo7gY/c5MUfuAf41gt5qnDaYuHYw49gOXyE+NmzMTVuXK77/Jr4K//e8G/+UfsfTA1p\niSHNWTrZeDCVtX8lM6pHY8KDyjHCXgghysDTXl5FeTQaTim1Cqjt5tSkMn2YUnE4q7a+c3N6DrBO\na72+hOuHA8MB6tevX5aP9imtNYn/mUzuxo3ETXuJkC6dS7/Ijb1pe3l0zaMk1EjgtatnEjC3B9Tp\ngG7Si+nv/EJsDRNDr0zwat6FEMKd8gSUrzxJpLXueb5zSqlTSqk4rXWiK2AklXCrwcCycxvdlVL/\nAWKAEaXkYx4wD6BTp07nLhJWaU6//TYZy5YRPWYMNW+9tVz3OJlzktGrRhPsF8yc6+dQY/dXkH4Y\n+s3g+91J/HYknZcGtiHQv+wN/EIIUVZlrvLSWj/jhc9dAdzn2r4PWF5C2rs4p7pLKfUA0Bu4S2vt\n8EJ+KlTG//7H6dlvEH7LzUSPHVOue2RbshmzegzZ1mzm9JxDXFA0rJsBdTpgb9yLGd/9RaPoEAZ1\nrLbNS0KIasbTySGzlFKZ57yOKqWWKaUaleNzpwG9lFL7gF6ufZRSnZRS84t8bgJQDzh3bft3gFjg\nF6XUdqXUs+XIQ6XI2bSJxImTCL7iCuKef75c82lZ7VbGrR3HgfQDvNLjFZpHNoffP3aWTno8zee/\nHWdfUjbjezfHT9Y5EUJUEE+rvF4BTuBcAljhbCSvDfyFc+LIHmX5UNca9Ne7Ob6FIl2AtdaHgGLr\n0mqty1NVV+nMBw5y7KGH8a9Xj/g3ZqMCAsp8D601k3+ZzMbEjUztNpUr61wJdqurdHI5+QnX8+qs\nH2kbH07f1u6asIQQwjc8/fnaR2s9V2udpbXOdLVL9NNaLwUifJi/i4YtJYWjI0ag/PyoN28uxvDy\nzf7/5vY3WfH3Csa0H8MtTW5xHixSOlny6xFOZOQzoU8LmU1YCFGhPA0oDqXUYKWUwfUaXORclWno\nrqoc+fkcGz0GW1IS9ea8RUB8+do1/m/v/zFvxzxua3obI9q6+iIUKZ1k1uvBW2v2c1XTaLo1Kd94\nFiGEKC9PA8rdOMd7JAGnXNtDlFJBwFgf5e2ioB0OTjw5gbwdO6gz42WC2rUr133WHVvHCxtfoHvd\n7jzT5ZkzpY8ipZP56w+Slmvlyd4tvPgEQgjhGU8nhzwA3HSe0z95LzsXn6RZs8hauZJaEyZQ44Yb\nynWPP07/wfgfx9MsohmzrpmFn8H1ZytSOkmufTXzl6ylf9s42sTLYppCiIrnaS+vZkqp1UqpXa79\ntkopb3QfvqilffwxqQveI+KfdxE59L7SL3DjWNYxRq8eTWRgJHN6ziHYP/jMySKlkzfX7Mdsc/B4\nr2Zeyr0QQpSNp1Ve7wJP45zdF631Dpw9vcR5ZK9bx8kpzxNyzdXETpxYrgby9Px0Rq0ahc1hY07P\nOUQHFWkXKVI6ORLZjQ83OaenbxQT6sWnEEIIz3na/TZYa73pnC9Fmw/yc1HI372b44+Ow9SiOfGv\nvILyK3sv53xbPg+veZgT2Sd494Z3aRR+znCf3z9yjYqfySur9mI0KB65vqmXnkAIIcrO0xLKaaVU\nY1w9upRStwOJPstVNWY9eZKjI0ZiqFGDem+/gyEkpMz3cGgHE3+ayPak7bx41YtcHnv52QmKlE7+\nDOnM8t/PTE8vhBCVxdOfzmNwzofVQil1HDiIs+eXKMKenc3RESNx5OTQ4MMP8I8tbRJl92ZsnsH3\nh7/niU5P0Duhd/EEv38E6Ueg3yxmrPyLMJMfI68u30zFQgjhLZ4GlOPAQmANEAlk4pyDa4qP8lXt\naJuN4+Mew7x/P/XmziWwefNy3ef9P95nye4lDLlsCPe2urd4giKlk1+Nl7Pmr195qm8LwoNlenoh\nROXyNKAsB9KBbTinYBFFaK05+fxUctavp/aU5wjt3q1c91l5aCUzt8ykV4NejO90nlUVXaUT3W8m\n07/7i9gaJu7rmlD+zAshhJd4GlDitdZ9fJqTaiz1vfdIX7qUqAcfJGLw4NIvcGPbqW08vf5p2sW0\n48XuL2I0uJlyvkjpZJW1HduObOXFAW0ICpDp6YUQlc/TRvmflVJtfJqTairz229JmjGTGv36EjPu\n0XLd40DGAR5e8zB1QuvwxnVvEOh3nsZ1V+nEfs1TzFjpnJ5+cCeZnl4IUTV4GlC6A1uVUn8ppXYo\npXYqpXb4MmPVQe623zjx5ASCOnQg7qWXUIayTxV/Ou80o1eNxqiMzOk5h5qBNd0nLCid1O3IsqyW\n7D2VzeM3yPT0Qoiqw9Mqr74+zUU1ZDlyhGNjxuAXV5v4OW9hMJnKfI9cay5jVo8hNT+Vhb0XUi+s\n3vkTu0onlt4zeHX5PtrUDadfG5meXghRdXg6l9dhX2ekOrGlpXF0+AhwOKg/dy5+EWWfwd/msDH+\nx/HsSd3DG9e9QavoViUkthSWTv57uhnH03cz/ba2Mj29EKJKkfqSMnJYLBx76CGsx48TP+ctAhIS\nynwPrTVTN05l/fH1PNPlGa6Ov7rkC1ylk9wrn+CttX/TvUk03ZvK9PRCiKpFAkoZaK1JnDiJvC1b\niZv2EsEdO5brPvN2zOOzfZ/xYJsHGdRsUMmJbRZYPxPqduSd441IzbHwZJ/yjXERQghfkoBSBsmz\nZ5P55ZfEPPoo4f37l+seK/5ewZvb3+SmRjfxUIeHSr/AVTrJ6Dye+T8dpH+bONrGn6fhXgghKlG1\nXJu9oh2+515syclYDh0i/PbbiBoxvFz3+fnEz/xnw3/oHNeZ5658rvQ2kCKlk1cP1sdsO8LjN8j0\n9EKIqklKKB6wZ2RgOXyYkCuvJO4//ylXY/hfqX/x2NrHaFizIa/2eBV/owdTpbhKJ0mXj+ODTUcY\n3EmmpxdCVF0SUEqhtcaamIgKDKTu66+h/Ms+Z9bJnJOMXjWaEP8Q5lw/h7CAsNIvKlI6eWlfPAYl\n09MLIao2qfIqhVIKU9OmaLsdY5gHgeAcmZZMRq0aRa4tl8V9F1M7xMOxI67SyZGuU/niixOMuLox\ntcNlenohRNUlAcUDCR9+UK7rLHYLj655lEOZh3in5zs0i/Cw/aNI6WTyn3GEmdIYdY1MTy+EqNqk\nystHHNrBvzf8m80nNzPlyil0juvs+cWu0slfLcbyw1/JjOrRRKanF0JUeRJQfGT2ttl8ffBrHrn8\nEW5qfJPnF9ossG4mum4nJu6MJbaGiaFXJvgsn0II4S2VElCUUpFKqe+VUvtc78XmLlFKXauU2l7k\nla+UuvWcNG8opbIrLueeWbpnKQt2LWBQs0Hc3/r+sl38+4eQcYTtjUay9Ug6j1zfTKanF0JUC5VV\nQnkKWK21bgqsdu2fRWu9RmvdXmvdHrgOyAVWFpxXSnUCqtwIvzVH1vDiphe5Jv4aJnaeWLYuxjYL\nrJuFrtuJCb/H0DA6hEEyPb0QopqorIByC7DYtb0YuLWEtAC3A99orXMBlFJGYAbwpM9yWMSwb4cx\n7NthpabbmbyTJ9c9ScvIlrx89cv4GcrY58FVOtkQ/wB7k3IYf0Nz/GV6eiFENVFZ31axWutEANd7\nrVLS3wl8VGR/LLCi4B4lUUoNV0ptUUptSU5OLneGS3M08yhjfxhLdFA0b1z/BsH+wWW7gat04qjb\nkQnba9Gmbjh9W8v09EKI6sNn3YaVUqsAd9+Ik8p4nzigDfCda78OMAjo4cn1Wut5wDyATp066bJ8\ntqdS81MZuWokDu3g7Z5vEx1UjpmAXaWT7xtO4Pjf+Uy7vS0Gg0xPL4SoPnwWULTWPc93Til1SikV\np7VOdAWMpBJuNRhYprW2uvY7AE2A/a72iWCl1H6tdRNv5b0s8mx5PPTDQ5zKPcX8G+aTEJ5Q9pu4\nSif2Oh15ekctujWpwVVNY7yeVyGE8KXKqvJaAdzn2r4PWF5C2rsoUt2ltf5Ka11ba52gtU4Acisr\nmNgddp5a9xQ7k3cy7apptK/Vvnw3cpVOlte8l9RcK0/2buHdjAohRAWorIAyDeillNoH9HLto5Tq\npJSaX5BIKZUA1AN+rIQ8lkhrzfTN0/nh6A9MuGICPRuct0BWMlfpxBrXkX/viqVfm9q0q1flOq8J\nIUSpKmXqFa11CnC9m+NbgAeK7B8C6pZyr0qZfnfxH4v5aM9H3NfyPu6+7O7y38hVOlka/Sj5Ns3j\nN8jiWUKI6kn6pJbDNwe/YdbWWfRO6M1jnR4r/41co+LNsZfz3J7aDO4UT2OZnl4IUU1JQCmjzSc3\nM+mnSVxe63Je6P4CBnUB/wm3fwAZR1kccCcGZeCR62XxLCFE9SUBpQz+Tv+bR9Y8QnxYPLOvm43J\naCr/zWwWWD+LvFodeGl/XYZ2S5Dp6YUQ1ZoEFA9Z7BZGrRqFyWji7Z5vE24Kv7Abukonb6vBhJn8\nZXp6IUS1J+uheMDusLMvfR8Ai/osom5oif0ESucqnWTHdGD24fo82acxNYMDvJBTIYSoPFJCKYXW\nmgMZB8iz5fFKj1doGdXywm/qKp28ar2NWmGBDLuy4YXfUwghKpmUUEqhlCImOIaIwAi61+1+4Td0\nlU4yotqz4HhDXhjQVITGr7sAAAsKSURBVKanF0JcFCSgeKCmyYsDDV2lk+nBD9AwOpTBnep5795C\nCFGJpMqrIrlKJykR7fgwtQmP39BMpqcXQlw05NusIrlKJ1Ozb6F13XD6tY6r7BwJIYTXSJVXRXGV\nTpLC27LsVHP+O7iFTE8vhLioSAmlomxfAhlHmZJ1M1c2jqZ7k3KsmSKEEFWYlFAqgmtG4cSwNnyZ\nfBnL+7Qo21rzQghRDUgJpSJsXwKZx/hPxo30bR0n09MLIS5KUkLxNVfp5GhIK1antWFlb5meXghx\ncZISiq8Vlk5uZlDHejI9vRDioiUlFF9ylU4OBrViQ1Zb1vZsWtk5EkIIn5ESii8Vlk5uYuiVDYkL\nD6rsHAkhhM9ICcUDC/ssLPtFrtLJflNLfqMDs3vI9PRCiIublFB8xVU6eS7rZkZe00SmpxdCXPSk\nhOILNgt63Sz2+l/GHr9OzO2WUNk5EkIIn5MSii/89l9U5jGm5tzCIz2bERwgcVsIcfGTbzpvs5nR\n619ht7EF/9/evcdIVZ9hHP8+IGiBequAiNQbrFCrYoKXBrRaRVFRUKxATaNtTGubVpvUFKtNjRpr\nW6RtYtJEqrQKFntB1HppvYCiSS2u3BVRNLWC3KpRpMDCMG//mB/pSpfd2dkze2Z3n0+y2XNmz5l5\nftnLu79zzrzn3QNPZeLJbk9vZl2DZyhZWzwLbV7DHdvG8/3zhro9vZl1GZ6hZKnQQLwwjRXdjuWD\n/iO58Hi3pzezrsP/Pmdp8Sy0eS0/334JU84f5vb0Ztal5FJQJB0s6WlJb6bPBzWxzVmSljT62C5p\nfPqaJN0u6Q1JKyVd2/6j2EOhgeIL01hKHYUjzuT0IW5Pb2ZdS14zlBuAZyNiCPBsWv+EiJgfEcMj\nYjjwJWAr8FT68lXAIGBoRAwDHmyX1M1ZPItum9dy545LmXLBMLenN7MuJ6+CMg64Ly3fB4xvYfvL\ngCcjYmta/xZwa0QUASJiY1VSlqvQwK4F01gcdfQeOprhbk9vZl1QXgWlf0SsA0if+7Ww/SRgdqP1\nY4CJkuolPSlpr10XJX0jbVe/adOmNgdv0uJZdP94Lb/cOYHrxwytzmuYmdW4ql3lJekZ4NAmvnRT\nK59nAHA88LdGD+8LbI+IEZIuBWYApze1f0RMB6YDjBgxIlrz2mUpNFB4/k6WFusYcNL5DO7n9vRm\n1jVVraBExDl7+5qkDZIGRMS6VDCaO2R1OTA3InY2emwNMCctzwUq6N6YkcUz2WfLe9wVN3LHuXW5\nxTAzy1teh7weBa5My1cCjzSz7WQ+ebgL4GFKJ+oBvgi8kWm6chUa2PncndQX66g77SK3pzezLi2v\ngvJTYLSkN4HRaR1JIyTds3sjSUdSuprr+Sb2nyBpOXAHcHU7ZP5/i2fS4z/rmN7tcr591uBcIpiZ\n1Ypc3ikfEe8DZzfxeD2NikNE/BMY2MR2HwIXVjFiywoN7Jh/J8uKdZx45ni3pzezLs/vlK9QLJpJ\nz63r+G2PSXxt1FF5xzEzy517eVWi0EDD/KmsKNZx2nkT3J7ezAzPUCpSXDST/batZ3avrzDplM/m\nHcfMrCb4X+vWKjSwfd5UXivW8cUxl7s9vZlZ4r+GrVR45X56bV/P3AO+ytgTDss7jplZzfAMpTUK\nDTTMm8qSYh3njp3k9vRmZo14htIKDQt/R++GDfy171WcUdc37zhmZjXFM5RypXfFLyvWceHFk92e\n3sxsD56hlGnL32fQZ8dGXhx4NScdcXDecczMao4LSjkKDRQXTKO+WMdF4yfnncbMrCa5oJRh1U9O\nY/+dm1h09DUM7r9/3nHMzGqSz6GUYeeuoD6GMHacZydmZnvjglKG7/aeRp/iR/zloF55RzEzq1ku\nKGXod0AvwMXEzKw5Lihl+MM3v5B3BDOzmueT8mZmlgkXFDMzy4QLipmZZcIFxczMMuGCYmZmmXBB\nMTOzTLigmJlZJlxQzMwsEy4oZmaWCUVE3hnajaRNwDsV7n4I8O8M4+Sps4yls4wDPJZa1VnG0tZx\nHBERLd6mtksVlLaQVB8RI/LOkYXOMpbOMg7wWGpVZxlLe43Dh7zMzCwTLihmZpYJF5TyTc87QIY6\ny1g6yzjAY6lVnWUs7TIOn0MxM7NMeIZiZmaZcEFpBUm3SVomaYmkpyQdlnemSkmaKun1NJ65kg7M\nO1MlJH1Z0quSipI65NU4ksZIWiVptaQb8s5TKUkzJG2UtCLvLG0haZCk+ZJWpp+t6/LOVClJ+0la\nKGlpGsstVX09H/Iqn6T9I2JzWr4W+FxEXJNzrIpIOheYFxEFST8DiIgpOcdqNUnDgCJwN3B9RNTn\nHKlVJHUH3gBGA2uAl4HJEfFarsEqIOkMYAtwf0R8Pu88lZI0ABgQEYskfRp4BRjfQb8nAnpHxBZJ\nPYAXgesi4qVqvJ5nKK2wu5gkvYEOW40j4qmIKKTVl4DD88xTqYhYGRGr8s7RBqcAqyPi7YjYATwI\njMs5U0UiYgHwQd452ioi1kXEorT8MbASGJhvqspEyZa02iN9VO3vlgtKK0m6XdK7wBXAj/POk5Gv\nA0/mHaKLGgi822h9DR30j1dnJOlI4CTgH/kmqZyk7pKWABuBpyOiamNxQdmDpGckrWjiYxxARNwU\nEYOAB4Dv5Ju2eS2NJW1zE1CgNJ6aVM44OjA18ViHnfl2JpL6AHOA7+1xdKJDiYhdETGc0lGIUyRV\n7XDkPtV64o4qIs4pc9PfA48DN1cxTpu0NBZJVwJjgbOjhk+mteJ70hGtAQY1Wj8ceC+nLJak8w1z\ngAci4qG882QhIj6U9BwwBqjKhROeobSCpCGNVi8GXs8rS1tJGgNMAS6OiK155+nCXgaGSDpKUk9g\nEvBozpm6tHQi+15gZUT8Iu88bSGp7+4rOCV9CjiHKv7d8lVerSBpDnAspauK3gGuiYi1+aaqjKTV\nwL7A++mhlzriFWuSLgHuAvoCHwJLIuK8fFO1jqQLgF8B3YEZEXF7zpEqImk2cCalzrYbgJsj4t5c\nQ1VA0ijgBWA5pd91gBsj4on8UlVG0gnAfZR+troBf4yIW6v2ei4oZmaWBR/yMjOzTLigmJlZJlxQ\nzMwsEy4oZmaWCRcUMzPLhAuKWYYkbWl5q2b3/7Oko9NyH0l3S3ordYpdIOlUST3Tst+YbDXFBcWs\nRkg6DugeEW+nh+6h1GxxSEQcB1wFHJKaSD4LTMwlqNleuKCYVYFKpqaeY8slTUyPd5P06zTjeEzS\nE5IuS7tdATyStjsGOBX4UUQUAVJH4sfTtg+n7c1qhqfMZtVxKTAcOJHSO8dflrQAGAkcCRwP9KPU\nGn1G2mckMDstH0fpXf+79vL8K4CTq5LcrEKeoZhVxyhgdur0ugF4nlIBGAX8KSKKEbEemN9onwHA\npnKePBWaHekGUGY1wQXFrDqaakvf3OMA24D90vKrwImSmvsd3RfYXkE2s6pwQTGrjgXAxHRzo77A\nGcBCSrdgnZDOpfSn1Exxt5XAYICIeAuoB25J3W+RNGT3PWAkfQbYFBE722tAZi1xQTGrjrnAMmAp\nMA/4QTrENYfSPVBWAHdTuhPgR2mfx/lkgbkaOBRYLWk58Bv+d6+Us4AO1/3WOjd3GzZrZ5L6RMSW\nNMtYCIyMiPXpfhXz0/reTsbvfo6HgB9GxKp2iGxWFl/lZdb+Hks3PeoJ3JZmLkTENkk3U7qn/L/2\ntnO6EdfDLiZWazxDMTOzTPgcipmZZcIFxczMMuGCYmZmmXBBMTOzTLigmJlZJlxQzMwsE/8F2HgT\nhwH3y8cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1c364cf8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#pd.DataFrame(grid.cv_results_).to_csv('LogisticGridSearchCV_Otto.csv')\n",
    "#cvresult = pd.DataFrame.from_csv('LogisticGridSearchCV_Otto.csv')\n",
    "#test_means = cv_results['mean_test_score']\n",
    "#test_stds = cv_results['std_test_score'] \n",
    "#train_means = cvresult['mean_train_score']\n",
    "#train_stds = cvresult['std_train_score'] \n",
    "\n",
    "\n",
    "# plot CV误差曲线\n",
    "test_means = grid.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "\n",
    "# plot results\n",
    "n_Cs = len(Cs)\n",
    "number_penaltys = len(penaltys)\n",
    "test_scores = np.array(test_means).reshape(n_Cs,number_penaltys)\n",
    "train_scores = np.array(train_means).reshape(n_Cs,number_penaltys)\n",
    "test_stds = np.array(test_stds).reshape(n_Cs,number_penaltys)\n",
    "train_stds = np.array(train_stds).reshape(n_Cs,number_penaltys)\n",
    "\n",
    "x_axis = np.log10(Cs)\n",
    "for i, value in enumerate(penaltys):\n",
    "    #pyplot.plot(log(Cs), test_scores[i], label= 'penalty:'   + str(value))\n",
    "    pyplot.errorbar(x_axis, test_scores[:,i], yerr=test_stds[:,i] ,label = penaltys[i] +' Test')\n",
    "    pyplot.errorbar(x_axis, train_scores[:,i], yerr=train_stds[:,i] ,label = penaltys[i] +' Train')\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'log(C)' )                                                                                                      \n",
    "pyplot.ylabel( 'neg-logloss' )\n",
    "pyplot.savefig('LogisticGridSearchCV_C.png' )\n",
    "\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上图给出了L1正则和L2正则下、不同正则参数C对应的模型在训练集上测试集上的正确率（score）。可以看出在训练集上C越大（正则越少）的模型性能越好；但在测试集上当C=100时性能最好（L1正则和L2正则均是）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成提交测试结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_test_pred = LR.predict_proba(X_test)\n",
    "out_df1 = pd.DataFrame(y_test_pred)\n",
    "out_df1.columns = [\"high\", \"medium\", \"low\"]\n",
    "\n",
    "out_df = pd.concat([test_Id,out_df1], axis = 1)\n",
    "out_df.to_csv(\"LR_Rent.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 74659 entries, 0 to 74658\n",
      "Data columns (total 1 columns):\n",
      "listing_id    74659 non-null int64\n",
      "dtypes: int64(1)\n",
      "memory usage: 583.3 KB\n"
     ]
    }
   ],
   "source": [
    "out_df = pd.concat([test_Id],axis = 1)\n",
    "out_df.to_csv(\"id.csv\",index=False)\n",
    "out_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
